You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/15 00:26:14 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6138 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6138/
Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 85 from http://127.0.0.1:60754/ftsc/o/collection1lastClient and got 83 from http://127.0.0.1:50472/ftsc/o/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 85 from http://127.0.0.1:60754/ftsc/o/collection1lastClient and got 83 from http://127.0.0.1:50472/ftsc/o/collection1
at __randomizedtesting.SeedInfo.seed([639C423C9375AEF3:E27ACC24E42ACECF]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9251 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 18031 T46 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ftsc/o
[junit4:junit4] 2> 18043 T46 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371247452987
[junit4:junit4] 2> 18046 T46 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 18049 T47 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 18149 T46 oasc.ZkTestServer.run start zk server on port:53898
[junit4:junit4] 2> 18191 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 18430 T53 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c004572 name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 18430 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 18432 T46 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 18457 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 18459 T55 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5eb09153 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 18460 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 18462 T46 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 18466 T46 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 18470 T46 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 18473 T46 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 18477 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 18486 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 18493 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 18494 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 18499 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 18499 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 18503 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 18504 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 18508 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 18508 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 18513 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 18513 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 18517 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 18517 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 18522 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 18522 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 18526 T46 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 18527 T46 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 18737 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 18822 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 18836 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 18837 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 18837 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475
[junit4:junit4] 2> 18837 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/solr.xml
[junit4:junit4] 2> 18838 T46 oasc.CoreContainer.<init> New CoreContainer 869373558
[junit4:junit4] 2> 18838 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/'
[junit4:junit4] 2> 18842 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/'
[junit4:junit4] 2> 18939 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 18940 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 18940 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 18941 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 18941 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 18942 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 18942 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 18942 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 18943 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 18955 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 19218 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 19219 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 19229 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 19230 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 19233 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e217f8f name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 19234 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 19238 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 19246 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 19249 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@797084b1 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 19249 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 19255 T46 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 19264 T46 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 19275 T46 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 19278 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56554_ftsc%2Fo
[junit4:junit4] 2> 19283 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56554_ftsc%2Fo
[junit4:junit4] 2> 19288 T46 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 19305 T46 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 19310 T46 oasc.Overseer.start Overseer (id=89866073084264451-127.0.0.1:56554_ftsc%2Fo-n_0000000000) starting
[junit4:junit4] 2> 19324 T46 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 19337 T72 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 19338 T46 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 19343 T46 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 19347 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 19360 T71 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 19377 T73 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 19377 T73 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 20869 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20871 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56554_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56554/ftsc/o"}
[junit4:junit4] 2> 20872 T71 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 20883 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 20896 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 21380 T73 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/collection1
[junit4:junit4] 2> 21380 T73 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 21382 T73 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 21382 T73 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 21385 T73 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/collection1/'
[junit4:junit4] 2> 21386 T73 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/collection1/lib/README' to classloader
[junit4:junit4] 2> 21387 T73 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 21437 T73 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 21487 T73 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 21489 T73 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 21501 T73 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 22066 T73 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 22080 T73 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 22084 T73 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 22122 T73 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 22128 T73 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 22133 T73 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 22136 T73 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 22136 T73 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 22136 T73 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 22138 T73 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 22138 T73 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 22139 T73 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 22146 T73 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371247453475/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/control/data/
[junit4:junit4] 2> 22146 T73 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 22147 T73 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 22147 T73 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/control/data
[junit4:junit4] 2> 22148 T73 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/control/data/index/
[junit4:junit4] 2> 22148 T73 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 22149 T73 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/control/data/index
[junit4:junit4] 2> 22168 T73 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 22169 T73 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 22171 T73 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 22172 T73 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 22172 T73 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 22173 T73 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 22174 T73 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 22174 T73 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 22174 T73 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 22175 T73 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 22175 T73 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 22178 T73 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 22182 T73 oass.SolrIndexSearcher.<init> Opening Searcher@70307b0e main
[junit4:junit4] 2> 22183 T73 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 22183 T73 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 22187 T74 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70307b0e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 22190 T73 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 22190 T73 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56554/ftsc/o collection:control_collection shard:shard1
[junit4:junit4] 2> 22194 T73 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 22202 T73 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 22211 T73 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 22215 T73 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 22215 T73 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 22216 T73 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56554/ftsc/o/collection1/
[junit4:junit4] 2> 22217 T73 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 22218 T73 oasc.SyncStrategy.syncToMe http://127.0.0.1:56554/ftsc/o/collection1/ has no replicas
[junit4:junit4] 2> 22218 T73 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56554/ftsc/o/collection1/
[junit4:junit4] 2> 22219 T73 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 22403 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 22415 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 22428 T73 oasc.ZkController.register We are http://127.0.0.1:56554/ftsc/o/collection1/ and leader is http://127.0.0.1:56554/ftsc/o/collection1/
[junit4:junit4] 2> 22429 T73 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56554/ftsc/o
[junit4:junit4] 2> 22429 T73 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 22429 T73 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 22429 T73 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 22432 T73 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 22435 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 22436 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 22452 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 22459 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 22462 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 22463 T77 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d2a7fd6 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 22464 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 22465 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 22469 T46 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 22548 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 22551 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 22552 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 22552 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 22553 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413
[junit4:junit4] 2> 22553 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/solr.xml
[junit4:junit4] 2> 22553 T46 oasc.CoreContainer.<init> New CoreContainer 1553673971
[junit4:junit4] 2> 22554 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/'
[junit4:junit4] 2> 22554 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/'
[junit4:junit4] 2> 22609 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 22610 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 22611 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 22611 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 22612 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 22613 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 22613 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 22614 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 22615 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 22615 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 22623 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 22624 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 22625 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 22627 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 22630 T88 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4377e312 name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 22630 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 22633 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 22638 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 22640 T90 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18980fb7 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 22640 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 22646 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 23651 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49686_ftsc%2Fo
[junit4:junit4] 2> 23653 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49686_ftsc%2Fo
[junit4:junit4] 2> 23658 T90 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 23658 T70 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 23658 T77 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 23659 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 23668 T91 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 23669 T91 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 23923 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23924 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56554_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56554/ftsc/o"}
[junit4:junit4] 2> 24031 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49686_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49686/ftsc/o"}
[junit4:junit4] 2> 24031 T71 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 24032 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 24085 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 24085 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 24085 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 24670 T91 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/collection1
[junit4:junit4] 2> 24671 T91 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 24672 T91 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 24672 T91 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 24674 T91 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/collection1/'
[junit4:junit4] 2> 24675 T91 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/collection1/lib/README' to classloader
[junit4:junit4] 2> 24676 T91 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 24708 T91 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 24755 T91 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 24757 T91 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 24764 T91 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 25137 T91 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 25145 T91 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 25149 T91 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 25177 T91 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 25183 T91 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 25188 T91 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 25190 T91 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 25191 T91 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 25191 T91 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 25193 T91 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 25193 T91 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 25194 T91 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 25194 T91 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371247457413/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/
[junit4:junit4] 2> 25195 T91 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 25196 T91 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 25197 T91 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1
[junit4:junit4] 2> 25197 T91 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index/
[junit4:junit4] 2> 25198 T91 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 25198 T91 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index
[junit4:junit4] 2> 25202 T91 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 25203 T91 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 25206 T91 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 25206 T91 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 25207 T91 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 25208 T91 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 25208 T91 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 25209 T91 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 25209 T91 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 25210 T91 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 25210 T91 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 25213 T91 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 25217 T91 oass.SolrIndexSearcher.<init> Opening Searcher@38f2146d main
[junit4:junit4] 2> 25218 T91 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 25219 T91 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 25223 T92 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38f2146d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 25226 T91 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 25226 T91 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49686/ftsc/o collection:collection1 shard:shard1
[junit4:junit4] 2> 25227 T91 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 25236 T91 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 25239 T91 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 25240 T91 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 25240 T91 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49686/ftsc/o/collection1/
[junit4:junit4] 2> 25240 T91 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 25241 T91 oasc.SyncStrategy.syncToMe http://127.0.0.1:49686/ftsc/o/collection1/ has no replicas
[junit4:junit4] 2> 25241 T91 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49686/ftsc/o/collection1/
[junit4:junit4] 2> 25241 T91 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 25595 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25606 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25606 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25606 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25650 T91 oasc.ZkController.register We are http://127.0.0.1:49686/ftsc/o/collection1/ and leader is http://127.0.0.1:49686/ftsc/o/collection1/
[junit4:junit4] 2> 25650 T91 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49686/ftsc/o
[junit4:junit4] 2> 25651 T91 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 25651 T91 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 25651 T91 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 25654 T91 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25655 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 25656 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 25656 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 25730 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 25733 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 25735 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 25735 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 25736 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603
[junit4:junit4] 2> 25737 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/solr.xml
[junit4:junit4] 2> 25737 T46 oasc.CoreContainer.<init> New CoreContainer 1862193356
[junit4:junit4] 2> 25738 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/'
[junit4:junit4] 2> 25739 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/'
[junit4:junit4] 2> 25804 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 25805 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 25805 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 25805 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 25806 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 25806 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 25807 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 25807 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 25807 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 25808 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 25812 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 25813 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 25813 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 25814 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 25816 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54bf066e name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 25816 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 25819 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 25823 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 25825 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71ce70e6 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 25825 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 25831 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 26836 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41494_ftsc%2Fo
[junit4:junit4] 2> 26838 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41494_ftsc%2Fo
[junit4:junit4] 2> 26843 T70 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 26843 T90 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 26844 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 26843 T77 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 26844 T106 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 26844 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 26845 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 26854 T107 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 26854 T107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 27115 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 27116 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49686_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49686/ftsc/o"}
[junit4:junit4] 2> 27122 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41494_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41494/ftsc/o"}
[junit4:junit4] 2> 27122 T71 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 27123 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 27131 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 27131 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 27131 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 27131 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 27857 T107 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/collection1
[junit4:junit4] 2> 27858 T107 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 27859 T107 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 27860 T107 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 27862 T107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/collection1/'
[junit4:junit4] 2> 27863 T107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/collection1/lib/README' to classloader
[junit4:junit4] 2> 27864 T107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 27909 T107 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 27985 T107 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 27987 T107 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 27995 T107 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 28499 T107 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 28506 T107 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 28509 T107 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 28534 T107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 28538 T107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 28543 T107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 28544 T107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 28545 T107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 28545 T107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 28547 T107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 28547 T107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 28547 T107 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 28548 T107 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371247460603/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/
[junit4:junit4] 2> 28548 T107 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 28549 T107 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 28549 T107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2
[junit4:junit4] 2> 28550 T107 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index/
[junit4:junit4] 2> 28550 T107 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 28550 T107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index
[junit4:junit4] 2> 28554 T107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 28554 T107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 28558 T107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 28558 T107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 28559 T107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 28560 T107 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 28560 T107 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 28561 T107 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 28561 T107 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 28562 T107 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 28562 T107 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 28565 T107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 28568 T107 oass.SolrIndexSearcher.<init> Opening Searcher@65998c51 main
[junit4:junit4] 2> 28569 T107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 28569 T107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 28574 T108 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@65998c51 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 28577 T107 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 28577 T107 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41494/ftsc/o collection:collection1 shard:shard2
[junit4:junit4] 2> 28578 T107 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 28586 T107 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 28589 T107 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 28589 T107 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 28590 T107 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41494/ftsc/o/collection1/
[junit4:junit4] 2> 28590 T107 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 28590 T107 oasc.SyncStrategy.syncToMe http://127.0.0.1:41494/ftsc/o/collection1/ has no replicas
[junit4:junit4] 2> 28590 T107 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41494/ftsc/o/collection1/
[junit4:junit4] 2> 28591 T107 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 28642 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 28654 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28654 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28654 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28654 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28697 T107 oasc.ZkController.register We are http://127.0.0.1:41494/ftsc/o/collection1/ and leader is http://127.0.0.1:41494/ftsc/o/collection1/
[junit4:junit4] 2> 28698 T107 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41494/ftsc/o
[junit4:junit4] 2> 28698 T107 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 28699 T107 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 28699 T107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 28704 T107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 28706 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 28707 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 28708 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 28789 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 28792 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 28793 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 28793 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 28794 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658
[junit4:junit4] 2> 28794 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/solr.xml
[junit4:junit4] 2> 28795 T46 oasc.CoreContainer.<init> New CoreContainer 1316404593
[junit4:junit4] 2> 28795 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/'
[junit4:junit4] 2> 28796 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/'
[junit4:junit4] 2> 28852 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 28853 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 28853 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 28854 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 28854 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 28855 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 28855 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 28855 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 28856 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 28856 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 28860 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 28861 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 28862 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 28863 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 28865 T120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@732d0fda name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 28865 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 28876 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 28881 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 28883 T122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e70d828 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 28884 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 28890 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 29896 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40199_ftsc%2Fo
[junit4:junit4] 2> 29898 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40199_ftsc%2Fo
[junit4:junit4] 2> 29901 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 29902 T77 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 29902 T122 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 29903 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 29902 T90 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 29903 T70 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 29904 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 29904 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 29906 T106 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 29913 T123 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 29913 T123 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 30165 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 30166 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41494_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41494/ftsc/o"}
[junit4:junit4] 2> 30173 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40199_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40199/ftsc/o"}
[junit4:junit4] 2> 30173 T71 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 30174 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 30184 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 30184 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 30184 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 30184 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 30184 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 30915 T123 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/collection1
[junit4:junit4] 2> 30915 T123 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 30916 T123 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 30917 T123 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 30918 T123 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/collection1/'
[junit4:junit4] 2> 30919 T123 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/collection1/lib/README' to classloader
[junit4:junit4] 2> 30919 T123 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 30945 T123 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 30976 T123 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 30978 T123 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 30985 T123 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 31362 T123 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 31370 T123 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 31373 T123 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 31399 T123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 31405 T123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 31411 T123 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 31413 T123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 31413 T123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 31414 T123 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 31415 T123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 31416 T123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 31416 T123 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 31417 T123 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371247463658/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/
[junit4:junit4] 2> 31417 T123 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 31418 T123 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 31419 T123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3
[junit4:junit4] 2> 31419 T123 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index/
[junit4:junit4] 2> 31420 T123 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 31420 T123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index
[junit4:junit4] 2> 31425 T123 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 31426 T123 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 31428 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 31429 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 31429 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 31430 T123 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 31430 T123 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 31431 T123 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 31431 T123 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 31432 T123 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 31432 T123 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 31435 T123 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 31438 T123 oass.SolrIndexSearcher.<init> Opening Searcher@589ff0b6 main
[junit4:junit4] 2> 31439 T123 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 31439 T123 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 31444 T124 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@589ff0b6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 31446 T123 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 31446 T123 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40199/ftsc/o collection:collection1 shard:shard3
[junit4:junit4] 2> 31447 T123 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 31456 T123 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 31459 T123 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 31460 T123 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 31460 T123 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40199/ftsc/o/collection1/
[junit4:junit4] 2> 31460 T123 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 31460 T123 oasc.SyncStrategy.syncToMe http://127.0.0.1:40199/ftsc/o/collection1/ has no replicas
[junit4:junit4] 2> 31461 T123 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40199/ftsc/o/collection1/
[junit4:junit4] 2> 31461 T123 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 31701 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 31713 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31713 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31713 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31713 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31713 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31796 T123 oasc.ZkController.register We are http://127.0.0.1:40199/ftsc/o/collection1/ and leader is http://127.0.0.1:40199/ftsc/o/collection1/
[junit4:junit4] 2> 31797 T123 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40199/ftsc/o
[junit4:junit4] 2> 31797 T123 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 31797 T123 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 31798 T123 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 31800 T123 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 31803 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 31804 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 31805 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 31894 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 31896 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 31897 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 31898 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 31898 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754
[junit4:junit4] 2> 31898 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/solr.xml
[junit4:junit4] 2> 31899 T46 oasc.CoreContainer.<init> New CoreContainer 993975022
[junit4:junit4] 2> 31899 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/'
[junit4:junit4] 2> 31900 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/'
[junit4:junit4] 2> 31948 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 31949 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 31949 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 31949 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 31950 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 31950 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 31951 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 31951 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 31951 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 31952 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 31956 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 31957 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 31957 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 31958 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 31960 T136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28894c51 name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 31960 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 31963 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 31967 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 31969 T138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ff09be3 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 31969 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 31974 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 32980 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35226_ftsc%2Fo
[junit4:junit4] 2> 32982 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35226_ftsc%2Fo
[junit4:junit4] 2> 32985 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 32985 T70 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 32986 T122 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 32986 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32985 T77 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 32987 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32988 T106 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 32986 T90 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 32986 T138 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 32986 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32990 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32998 T139 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 32998 T139 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 33254 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 33256 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40199_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40199/ftsc/o"}
[junit4:junit4] 2> 33261 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35226_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35226/ftsc/o"}
[junit4:junit4] 2> 33261 T71 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 33261 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 33267 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33267 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33267 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33267 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33267 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33267 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34000 T139 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/collection1
[junit4:junit4] 2> 34001 T139 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 34002 T139 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 34002 T139 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 34003 T139 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/collection1/'
[junit4:junit4] 2> 34004 T139 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/collection1/lib/README' to classloader
[junit4:junit4] 2> 34005 T139 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 34043 T139 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 34087 T139 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 34089 T139 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 34097 T139 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 34484 T139 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 34491 T139 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 34494 T139 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 34516 T139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 34520 T139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 34524 T139 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 34526 T139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 34526 T139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 34526 T139 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 34528 T139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 34528 T139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 34529 T139 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 34529 T139 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371247466754/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/
[junit4:junit4] 2> 34529 T139 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 34530 T139 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 34531 T139 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4
[junit4:junit4] 2> 34531 T139 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index/
[junit4:junit4] 2> 34531 T139 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 34532 T139 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index
[junit4:junit4] 2> 34535 T139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 34535 T139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 34537 T139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 34538 T139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 34538 T139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 34539 T139 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 34540 T139 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 34540 T139 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 34541 T139 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 34541 T139 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 34542 T139 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 34544 T139 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 34547 T139 oass.SolrIndexSearcher.<init> Opening Searcher@3fa2ec6a main
[junit4:junit4] 2> 34548 T139 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 34549 T139 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 34554 T140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fa2ec6a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 34557 T139 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 34557 T139 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35226/ftsc/o collection:collection1 shard:shard1
[junit4:junit4] 2> 34561 T139 oasc.ZkController.register We are http://127.0.0.1:35226/ftsc/o/collection1/ and leader is http://127.0.0.1:49686/ftsc/o/collection1/
[junit4:junit4] 2> 34561 T139 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35226/ftsc/o
[junit4:junit4] 2> 34561 T139 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 34561 T139 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C27 name=collection1 org.apache.solr.core.SolrCore@5f0b1da2 url=http://127.0.0.1:35226/ftsc/o/collection1 node=127.0.0.1:35226_ftsc%2Fo C27_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:35226_ftsc%2Fo, base_url=http://127.0.0.1:35226/ftsc/o}
[junit4:junit4] 2> 34564 T141 C27 P35226 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 34565 T139 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 34565 T141 C27 P35226 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 34566 T141 C27 P35226 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 34566 T141 C27 P35226 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 34566 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 34567 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 34567 T141 C27 P35226 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 34567 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 34662 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 34664 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 34665 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 34665 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 34666 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517
[junit4:junit4] 2> 34666 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/solr.xml
[junit4:junit4] 2> 34666 T46 oasc.CoreContainer.<init> New CoreContainer 1288818343
[junit4:junit4] 2> 34667 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/'
[junit4:junit4] 2> 34667 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/'
[junit4:junit4] 2> 34713 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 34714 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 34714 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 34714 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 34715 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 34715 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 34716 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 34716 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 34716 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 34717 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 34721 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 34722 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 34722 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 34723 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 34725 T153 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@784f6235 name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 34726 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 34728 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 34733 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 34734 T155 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@565b576f name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 34735 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 34740 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 34778 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 34780 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35226_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35226/ftsc/o"}
[junit4:junit4] 2> 34789 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34789 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34789 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34789 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34789 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34789 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34791 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34836 T86 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 34836 T86 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
[junit4:junit4] 2> 34837 T86 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:35226_ftsc%252Fo&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=3
[junit4:junit4] 2> 35745 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42394_ftsc%2Fo
[junit4:junit4] 2> 35747 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42394_ftsc%2Fo
[junit4:junit4] 2> 35750 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 35750 T155 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35751 T122 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35751 T70 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35752 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 35751 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 35751 T90 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35754 T138 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35751 T77 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35750 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 35755 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 35754 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 35752 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 35760 T106 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 35767 T156 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 35768 T156 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 36307 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 36309 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42394_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42394/ftsc/o"}
[junit4:junit4] 2> 36309 T71 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 36310 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 36316 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36316 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36317 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36316 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36316 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36316 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36317 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 36770 T156 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/collection1
[junit4:junit4] 2> 36771 T156 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 36772 T156 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 36772 T156 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 36774 T156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/collection1/'
[junit4:junit4] 2> 36775 T156 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/collection1/lib/README' to classloader
[junit4:junit4] 2> 36775 T156 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 36852 T156 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> ASYNC NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@5f0b1da2 url=http://127.0.0.1:35226/ftsc/o/collection1 node=127.0.0.1:35226_ftsc%2Fo C28_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35226_ftsc%2Fo, base_url=http://127.0.0.1:35226/ftsc/o}
[junit4:junit4] 2> 36876 T141 C28 P35226 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49686/ftsc/o/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 36879 T141 C28 P35226 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 36886 T141 C28 P35226 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35226/ftsc/o START replicas=[http://127.0.0.1:49686/ftsc/o/collection1/] nUpdates=100
[junit4:junit4] 2> 36889 T141 C28 P35226 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 36889 T141 C28 P35226 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 36889 T141 C28 P35226 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 36890 T141 C28 P35226 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 36890 T141 C28 P35226 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 36890 T141 C28 P35226 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49686/ftsc/o/collection1/. core=collection1
[junit4:junit4] 2> 36891 T141 C28 P35226 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 36900 T156 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 36908 T156 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> ASYNC NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@7f38870a url=http://127.0.0.1:49686/ftsc/o/collection1 node=127.0.0.1:49686_ftsc%2Fo C29_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49686_ftsc%2Fo, base_url=http://127.0.0.1:49686/ftsc/o, leader=true}
[junit4:junit4] 2> 36914 T86 C29 P49686 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 36919 T85 C29 P49686 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 36921 T86 C29 P49686 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 36924 T156 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 36932 T86 C29 P49686 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 36937 T86 C29 P49686 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 36937 T86 C29 P49686 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 36938 T86 C29 P49686 oass.SolrIndexSearcher.<init> Opening Searcher@490fbc83 realtime
[junit4:junit4] 2> 36939 T86 C29 P49686 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 36939 T86 C29 P49686 oasup.LogUpdateProcessor.finish [collection1] webapp=/ftsc/o path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4] 2> 36960 T141 C28 P35226 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 36960 T141 C28 P35226 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 36969 T141 C28 P35226 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 37001 T85 C29 P49686 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 37002 T85 C29 P49686 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=24
[junit4:junit4] 2> 37004 T141 C28 P35226 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 37005 T141 C28 P35226 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 37005 T141 C28 P35226 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 37011 T86 C29 P49686 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 37012 T141 C28 P35226 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 37014 T141 C28 P35226 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index.20130615080431956
[junit4:junit4] 2> 37014 T141 C28 P35226 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index.20130615080431956 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c648855 fullCopy=false
[junit4:junit4] 2> ASYNC NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@7f38870a url=http://127.0.0.1:49686/ftsc/o/collection1 node=127.0.0.1:49686_ftsc%2Fo C30_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49686_ftsc%2Fo, base_url=http://127.0.0.1:49686/ftsc/o, leader=true}
[junit4:junit4] 2> 37040 T84 C30 P49686 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=21
[junit4:junit4] 2> ASYNC NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@5f0b1da2 url=http://127.0.0.1:35226/ftsc/o/collection1 node=127.0.0.1:35226_ftsc%2Fo C31_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35226_ftsc%2Fo, base_url=http://127.0.0.1:35226/ftsc/o}
[junit4:junit4] 2> 37045 T141 C31 P35226 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 37062 T141 C31 P35226 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 37062 T141 C31 P35226 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 37066 T141 C31 P35226 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 37066 T141 C31 P35226 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 37067 T141 C31 P35226 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 37067 T141 C31 P35226 oass.SolrIndexSearcher.<init> Opening Searcher@1bc0c549 main
[junit4:junit4] 2> 37069 T140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bc0c549 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 37070 T141 C31 P35226 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index.20130615080431956 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index.20130615080431956;done=true>>]
[junit4:junit4] 2> 37070 T141 C31 P35226 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index.20130615080431956
[junit4:junit4] 2> 37071 T141 C31 P35226 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty4/index.20130615080431956
[junit4:junit4] 2> 37074 T141 C31 P35226 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 37074 T141 C31 P35226 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 37074 T141 C31 P35226 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 37075 T141 C31 P35226 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 37077 T141 C31 P35226 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 37353 T156 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 37360 T156 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 37363 T156 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 37386 T156 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 37393 T156 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 37397 T156 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 37399 T156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 37400 T156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 37400 T156 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 37402 T156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 37402 T156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 37403 T156 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 37403 T156 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371247469517/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/
[junit4:junit4] 2> 37403 T156 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 37404 T156 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 37405 T156 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5
[junit4:junit4] 2> 37405 T156 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index/
[junit4:junit4] 2> 37405 T156 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 37406 T156 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index
[junit4:junit4] 2> 37428 T156 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 37429 T156 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 37431 T156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 37432 T156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 37433 T156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 37434 T156 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 37434 T156 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 37435 T156 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 37440 T156 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 37441 T156 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 37441 T156 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 37444 T156 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 37448 T156 oass.SolrIndexSearcher.<init> Opening Searcher@4c2cffbe main
[junit4:junit4] 2> 37449 T156 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 37450 T156 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 37458 T156 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 37459 T156 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42394/ftsc/o collection:collection1 shard:shard2
[junit4:junit4] 2> 37463 T159 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c2cffbe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 37466 T156 oasc.ZkController.register We are http://127.0.0.1:42394/ftsc/o/collection1/ and leader is http://127.0.0.1:41494/ftsc/o/collection1/
[junit4:junit4] 2> 37466 T156 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42394/ftsc/o
[junit4:junit4] 2> 37466 T156 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 37467 T156 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@2c712142 url=http://127.0.0.1:42394/ftsc/o/collection1 node=127.0.0.1:42394_ftsc%2Fo C32_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:42394_ftsc%2Fo, base_url=http://127.0.0.1:42394/ftsc/o}
[junit4:junit4] 2> 37467 T160 C32 P42394 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 37468 T160 C32 P42394 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 37468 T156 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 37468 T160 C32 P42394 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 37468 T160 C32 P42394 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 37469 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 37470 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 37470 T160 C32 P42394 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 37470 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 37476 T102 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 37546 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 37549 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 37549 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 37550 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 37550 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417
[junit4:junit4] 2> 37551 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/solr.xml
[junit4:junit4] 2> 37551 T46 oasc.CoreContainer.<init> New CoreContainer 1216903373
[junit4:junit4] 2> 37551 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/'
[junit4:junit4] 2> 37552 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/'
[junit4:junit4] 2> 37599 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 37600 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 37600 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 37601 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 37601 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 37601 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 37602 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 37602 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 37603 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 37603 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 37607 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 37608 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 37608 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 37610 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 37611 T172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65f673ad name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 37612 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 37614 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 37618 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 37620 T174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131c8e88 name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 37620 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 37624 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 37824 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 37826 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35226_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35226/ftsc/o"}
[junit4:junit4] 2> 37830 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42394_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42394/ftsc/o"}
[junit4:junit4] 2> 37834 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37835 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37835 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37835 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37835 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37835 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37834 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37834 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 38477 T102 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 38478 T102 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:42394_ftsc%252Fo&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 38629 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60754_ftsc%2Fo
[junit4:junit4] 2> 38632 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60754_ftsc%2Fo
[junit4:junit4] 2> 38636 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 38637 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 38638 T155 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38638 T70 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38639 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 38639 T77 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38638 T122 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38638 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 38638 T90 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38640 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 38639 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 38639 T174 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38641 T106 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38640 T138 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 38640 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 38641 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 38646 T175 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 38647 T175 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 39346 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 39347 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60754_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60754/ftsc/o"}
[junit4:junit4] 2> 39348 T71 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 39348 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 39352 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39353 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39353 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39353 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39352 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39353 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39352 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39352 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39648 T175 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/collection1
[junit4:junit4] 2> 39649 T175 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 39650 T175 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 39650 T175 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 39652 T175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/collection1/'
[junit4:junit4] 2> 39652 T175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/collection1/lib/README' to classloader
[junit4:junit4] 2> 39653 T175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 39682 T175 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 39725 T175 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 39727 T175 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 39734 T175 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 40108 T175 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 40116 T175 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 40119 T175 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 40140 T175 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 40144 T175 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 40147 T175 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 40153 T175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 40154 T175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 40154 T175 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 40156 T175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 40156 T175 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 40156 T175 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 40157 T175 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371247472417/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/
[junit4:junit4] 2> 40157 T175 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fadbf85
[junit4:junit4] 2> 40157 T175 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 40158 T175 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6
[junit4:junit4] 2> 40158 T175 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index/
[junit4:junit4] 2> 40158 T175 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 40159 T175 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index
[junit4:junit4] 2> 40162 T175 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 40162 T175 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 40164 T175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 40165 T175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 40165 T175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 40166 T175 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 40166 T175 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 40167 T175 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 40167 T175 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 40168 T175 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 40168 T175 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 40171 T175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 40174 T175 oass.SolrIndexSearcher.<init> Opening Searcher@2c8dacaf main
[junit4:junit4] 2> 40174 T175 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 40175 T175 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 40180 T176 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c8dacaf main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 40182 T175 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 40183 T175 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60754/ftsc/o collection:collection1 shard:shard3
[junit4:junit4] 2> 40186 T175 oasc.ZkController.register We are http://127.0.0.1:60754/ftsc/o/collection1/ and leader is http://127.0.0.1:40199/ftsc/o/collection1/
[junit4:junit4] 2> 40186 T175 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60754/ftsc/o
[junit4:junit4] 2> 40186 T175 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 40187 T175 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C33 name=collection1 org.apache.solr.core.SolrCore@4575f798 url=http://127.0.0.1:60754/ftsc/o/collection1 node=127.0.0.1:60754_ftsc%2Fo C33_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:60754_ftsc%2Fo, base_url=http://127.0.0.1:60754/ftsc/o}
[junit4:junit4] 2> 40187 T177 C33 P60754 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 40188 T177 C33 P60754 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 40188 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 40188 T177 C33 P60754 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 40188 T177 C33 P60754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 40189 T46 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 40190 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 40190 T177 C33 P60754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 40190 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 40196 T116 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 40260 T46 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 40263 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4:junit4] 2> 40263 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 40264 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 40264 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371247475137
[junit4:junit4] 2> 40265 T46 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371247475137/solr.xml
[junit4:junit4] 2> 40265 T46 oasc.CoreContainer.<init> New CoreContainer 948092696
[junit4:junit4] 2> 40265 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371247475137/'
[junit4:junit4] 2> 40266 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371247475137/'
[junit4:junit4] 2> 40334 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 40335 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 40335 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 40336 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 40336 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 40336 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 40337 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 40337 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 40338 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 40339 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 40345 T46 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 40346 T46 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53898/solr
[junit4:junit4] 2> 40346 T46 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 40348 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 40349 T189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5808b672 name:ZooKeeperConnection Watcher:127.0.0.1:53898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 40350 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 40353 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 40358 T46 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 40360 T191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fa0d32f name:ZooKeeperConnection Watcher:127.0.0.1:53898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 40360 T46 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 40365 T46 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C34 name=collection1 org.apache.solr.core.SolrCore@2c712142 url=http://127.0.0.1:42394/ftsc/o/collection1 node=127.0.0.1:42394_ftsc%2Fo C34_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:42394_ftsc%2Fo, base_url=http://127.0.0.1:42394/ftsc/o}
[junit4:junit4] 2> 40479 T160 C34 P42394 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:41494/ftsc/o/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 40479 T160 C34 P42394 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42394/ftsc/o START replicas=[http://127.0.0.1:41494/ftsc/o/collection1/] nUpdates=100
[junit4:junit4] 2> 40480 T160 C34 P42394 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 40481 T160 C34 P42394 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 40481 T160 C34 P42394 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 40481 T160 C34 P42394 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 40482 T160 C34 P42394 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 40482 T160 C34 P42394 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:41494/ftsc/o/collection1/. core=collection1
[junit4:junit4] 2> 40483 T160 C34 P42394 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C35 name=collection1 org.apache.solr.core.SolrCore@551cbf36 url=http://127.0.0.1:41494/ftsc/o/collection1 node=127.0.0.1:41494_ftsc%2Fo C35_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41494_ftsc%2Fo, base_url=http://127.0.0.1:41494/ftsc/o, leader=true}
[junit4:junit4] 2> 40486 T101 C35 P41494 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 40489 T100 C35 P41494 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 40491 T100 C35 P41494 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 40492 T100 C35 P41494 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 40497 T100 C35 P41494 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 40497 T100 C35 P41494 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 40498 T100 C35 P41494 oass.SolrIndexSearcher.<init> Opening Searcher@400c420b realtime
[junit4:junit4] 2> 40498 T100 C35 P41494 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 40499 T100 C35 P41494 oasup.LogUpdateProcessor.finish [collection1] webapp=/ftsc/o path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 40500 T160 C34 P42394 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 40500 T160 C34 P42394 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 40503 T102 C35 P41494 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 40503 T102 C35 P41494 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 40504 T160 C34 P42394 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 40505 T160 C34 P42394 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 40505 T160 C34 P42394 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 40507 T101 C35 P41494 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 40508 T160 C34 P42394 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 40509 T160 C34 P42394 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index.20130615080435452
[junit4:junit4] 2> 40510 T160 C34 P42394 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index.20130615080435452 lockFactory=org.apache.lucene.store.NativeFSLockFactory@77f52c80 fullCopy=false
[junit4:junit4] 2> 40513 T100 C35 P41494 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 40514 T160 C34 P42394 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 40528 T160 C34 P42394 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 40528 T160 C34 P42394 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 40530 T160 C34 P42394 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 40531 T160 C34 P42394 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 40531 T160 C34 P42394 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 40532 T160 C34 P42394 oass.SolrIndexSearcher.<init> Opening Searcher@17249f80 main
[junit4:junit4] 2> 40533 T159 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17249f80 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 40533 T160 C34 P42394 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index.20130615080435452 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index.20130615080435452;done=true>>]
[junit4:junit4] 2> 40534 T160 C34 P42394 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index.20130615080435452
[junit4:junit4] 2> 40534 T160 C34 P42394 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty5/index.20130615080435452
[junit4:junit4] 2> 40534 T160 C34 P42394 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 40535 T160 C34 P42394 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 40535 T160 C34 P42394 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 40535 T160 C34 P42394 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 40537 T160 C34 P42394 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 40860 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 40861 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60754_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60754/ftsc/o"}
[junit4:junit4] 2> 40865 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42394_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42394/ftsc/o"}
[junit4:junit4] 2> 40887 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40888 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40888 T191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40888 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40888 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40888 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40888 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40887 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40887 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 41197 T116 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 41197 T116 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:60754_ftsc%252Fo&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 41370 T46 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56947_ftsc%2Fo
[junit4:junit4] 2> 41371 T46 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56947_ftsc%2Fo
[junit4:junit4] 2> 41375 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 41376 T70 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41377 T155 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41377 T122 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41377 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 41377 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 41377 T191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 41377 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 41377 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 41379 T174 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41379 T106 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41379 T77 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41378 T90 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41380 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 41380 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 41381 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 41384 T191 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41384 T138 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 41390 T194 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 41390 T194 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 42401 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 42402 T71 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56947_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56947/ftsc/o"}
[junit4:junit4] 2> 42403 T71 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 42403 T71 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 42408 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T138 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T77 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42408 T155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> ASYNC NEW_CORE C36 name=collection1 org.apache.solr.core.SolrCore@4575f798 url=http://127.0.0.1:60754/ftsc/o/collection1 node=127.0.0.1:60754_ftsc%2Fo C36_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:60754_ftsc%2Fo, base_url=http://127.0.0.1:60754/ftsc/o}
[junit4:junit4] 2> 43198 T177 C36 P60754 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40199/ftsc/o/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 43199 T177 C36 P60754 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60754/ftsc/o START replicas=[http://127.0.0.1:40199/ftsc/o/collection1/] nUpdates=100
[junit4:junit4] 2> 43199 T177 C36 P60754 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 43200 T177 C36 P60754 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 43200 T177 C36 P60754 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 43200 T177 C36 P60754 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 43201 T177 C36 P60754 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 43201 T177 C36 P60754 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40199/ftsc/o/collection1/. core=collection1
[junit4:junit4] 2> 43201 T177 C36 P60754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C37 name=collection1 org.apache.solr.core.SolrCore@4adf5232 url=http://127.0.0.1:40199/ftsc/o/collection1 node=127.0.0.1:40199_ftsc%2Fo C37_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:40199_ftsc%2Fo, base_url=http://127.0.0.1:40199/ftsc/o, leader=true}
[junit4:junit4] 2> 43207 T115 C37 P40199 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 43209 T118 C37 P40199 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 43213 T118 C37 P40199 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 43213 T118 C37 P40199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 43227 T118 C37 P40199 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 43228 T118 C37 P40199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 43229 T118 C37 P40199 oass.SolrIndexSearcher.<init> Opening Searcher@20a0a575 realtime
[junit4:junit4] 2> 43229 T118 C37 P40199 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 43230 T118 C37 P40199 oasup.LogUpdateProcessor.finish [collection1] webapp=/ftsc/o path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> 43231 T177 C36 P60754 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 43231 T177 C36 P60754 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 43233 T115 C37 P40199 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 43234 T115 C37 P40199 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 43235 T177 C36 P60754 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 43235 T177 C36 P60754 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 43235 T177 C36 P60754 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 43238 T118 C37 P40199 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 43239 T177 C36 P60754 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 43240 T177 C36 P60754 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index.20130615080438182
[junit4:junit4] 2> 43240 T177 C36 P60754 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index.20130615080438182 lockFactory=org.apache.lucene.store.NativeFSLockFactory@535ac0f0 fullCopy=false
[junit4:junit4] 2> 43243 T116 C37 P40199 oasc.SolrCore.execute [collection1] webapp=/ftsc/o path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 43245 T177 C36 P60754 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 43258 T177 C36 P60754 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 43258 T177 C36 P60754 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 43260 T177 C36 P60754 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 43261 T177 C36 P60754 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 43261 T177 C36 P60754 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 43262 T177 C36 P60754 oass.SolrIndexSearcher.<init> Opening Searcher@33a6270b main
[junit4:junit4] 2> 43263 T176 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33a6270b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 43263 T177 C36 P60754 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index.20130615080438182 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index.20130615080438182;done=true>>]
[junit4:junit4] 2> 43264 T177 C36 P60754 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index.20130615080438182
[junit4:junit4] 2> 43264 T177 C36 P60754 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty6/index.20130615080438182
[junit4:junit4] 2> 43264 T177 C36 P60754 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 43264 T177 C36 P60754 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 43265 T177 C36 P60754 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 43265 T177 C36 P60754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 43267 T177 C36 P60754 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 43392 T194 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371247475137/collection1
[junit4:junit4] 2> 43393 T194 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 43394 T194 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 43394 T194 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 43396 T194 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371247475137/collection1/'
[junit4:junit4] 2> 43396 T194 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspac
[...truncated too long message...]
cc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 188487 T467 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 188489 T467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 188490 T467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33279_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33279/ftsc/o"}
[junit4:junit4] 2> 188493 T467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "numShards":null,
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33279_ftsc%2Fo",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33279/ftsc/o"}
[junit4:junit4] 2> 188496 T288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 188502 T46 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/ftsc/o,null}
[junit4:junit4] 2> 188553 T46 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50472
[junit4:junit4] 2> 188554 T46 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=654836083
[junit4:junit4] 2> 188693 T248 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 188693 T248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 188693 T248 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 189556 T46 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 189557 T46 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 189558 T46 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@44807612
[junit4:junit4] 2> 189560 T46 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=174,cumulative_deletesById=91,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 189561 T46 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 189561 T46 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 189561 T46 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 189562 T46 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 189563 T46 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 189564 T46 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty12;done=false>>]
[junit4:junit4] 2> 189564 T46 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty12
[junit4:junit4] 2> 189564 T46 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty12/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty12/index;done=false>>]
[junit4:junit4] 2> 189565 T46 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371247452987/jetty12/index
[junit4:junit4] 2> 189565 T467 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89866073084264476-127.0.0.1:50472_ftsc%2Fo-n_0000000012) am no longer a leader.
[junit4:junit4] 2> 189589 T46 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/ftsc/o,null}
[junit4:junit4] 2> 189668 T46 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 189670 T46 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53898 53898
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=639C423C9375AEF3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hi_IN -Dtests.timezone=Asia/Yakutsk -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 172s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard3 is not consistent. Got 85 from http://127.0.0.1:60754/ftsc/o/collection1lastClient and got 83 from http://127.0.0.1:50472/ftsc/o/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([639C423C9375AEF3:E27ACC24E42ACECF]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 189741 T46 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 171714 T45 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 190867 T288 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 190867 T288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 190868 T288 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Direct), text=PostingsFormat(name=Memory doPackFST= false), _version_=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Asserting), intDefault=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Direct), id=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=Direct), range_facet_sl=PostingsFormat(name=Memory doPackFST= false), range_facet_si=MockVariableIntBlock(baseBlockSize=54), other_tl1=PostingsFormat(name=Direct), multiDefault=MockVariableIntBlock(baseBlockSize=54), a_si=MockVariableIntBlock(baseBlockSize=54)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=DefaultSimilarity, locale=hi_IN, timezone=Asia/Yakutsk
[junit4:junit4] 2> NOTE: Linux 3.2.0-45-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=66735616,total=137363456
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestBM25SimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestRecovery, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 173.01s, 1 test, 1 failure <<< FAILURES!
[...truncated 861 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:385: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:365: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 39 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6139 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6139/
Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<209> but was:<40>
Stack Trace:
java.lang.AssertionError: expected:<209> but was:<40>
at __randomizedtesting.SeedInfo.seed([A5AF8CA1B1A42A9C:244902B9C6FB4AA0]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9175 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
[junit4:junit4] 2> 868 T10 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 898 T10 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371252838077
[junit4:junit4] 2> 902 T10 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 905 T11 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 1005 T10 oasc.ZkTestServer.run start zk server on port:48085
[junit4:junit4] 2> 1056 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1752 T12 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
[junit4:junit4] 2> 3333 T17 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b36612 name:ZooKeeperConnection Watcher:127.0.0.1:48085 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3334 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3336 T10 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 3369 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3372 T19 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bd8bd9 name:ZooKeeperConnection Watcher:127.0.0.1:48085/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3372 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3380 T10 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 3387 T10 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 3402 T10 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 3410 T10 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 3419 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 3438 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 3447 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 3448 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 3454 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 3455 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 3459 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 3460 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 3465 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 3466 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 3470 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 3471 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 3476 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 3477 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 3482 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 3483 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 3489 T10 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 3490 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 3829 T10 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 3930 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56743
[junit4:junit4] 2> 3960 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 4011 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 4012 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675
[junit4:junit4] 2> 4026 T10 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/solr.xml
[junit4:junit4] 2> 4033 T10 oasc.CoreContainer.<init> New CoreContainer 10116488
[junit4:junit4] 2> 4034 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/'
[junit4:junit4] 2> 4035 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/'
[junit4:junit4] 2> 4388 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 4389 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 4390 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 4391 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 4391 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 4392 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 4393 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 4394 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 4395 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 4420 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 4701 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 4703 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48085/solr
[junit4:junit4] 2> 4713 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 4715 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 4719 T32 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@199e3fe name:ZooKeeperConnection Watcher:127.0.0.1:48085 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 4719 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 4723 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 4732 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 4734 T34 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@186dead name:ZooKeeperConnection Watcher:127.0.0.1:48085/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 4734 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 4739 T10 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 4747 T10 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 4757 T10 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 4760 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56743_
[junit4:junit4] 2> 4763 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56743_
[junit4:junit4] 2> 4769 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 4784 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 4789 T10 oasc.Overseer.start Overseer (id=89866426002964483-127.0.0.1:56743_-n_0000000000) starting
[junit4:junit4] 2> 4801 T10 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 4812 T36 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 4812 T10 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 4818 T10 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 4821 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4831 T35 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 4864 T37 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 4865 T37 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6339 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6341 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:56743_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56743",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 6342 T35 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 6351 T35 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 6364 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 6868 T37 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/collection1
[junit4:junit4] 2> 6869 T37 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 6871 T37 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 6871 T37 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 6875 T37 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/collection1/'
[junit4:junit4] 2> 6877 T37 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/collection1/lib/README' to classloader
[junit4:junit4] 2> 6877 T37 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 6976 T37 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 7168 T37 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 7182 T37 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 7262 T37 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 8197 T37 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 8220 T37 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 8223 T37 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 8405 T37 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 8413 T37 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 8419 T37 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 8422 T37 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 8422 T37 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 8423 T37 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 8425 T37 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 8425 T37 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 8454 T37 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 8463 T37 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371252840675/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/control/data/
[junit4:junit4] 2> 8466 T37 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@614cee
[junit4:junit4] 2> 8538 T37 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 8568 T37 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/control/data
[junit4:junit4] 2> 8569 T37 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/control/data/index/
[junit4:junit4] 2> 8569 T37 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371252838077/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 8581 T37 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/control/data/index
[junit4:junit4] 2> 8684 T37 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368ebd lockFactory=org.apache.lucene.store.NativeFSLockFactory@16988ac),segFN=segments_1,generation=1}
[junit4:junit4] 2> 8685 T37 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 8883 T37 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 8883 T37 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 8907 T37 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 8926 T37 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 8933 T37 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 8934 T37 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 8936 T37 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 8949 T37 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 8952 T37 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 8993 T37 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9022 T37 oass.SolrIndexSearcher.<init> Opening Searcher@a9d2a7 main
[junit4:junit4] 2> 9043 T37 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/control/data/tlog
[junit4:junit4] 2> 9050 T37 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9051 T37 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9102 T38 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a9d2a7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9107 T37 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 9108 T37 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56743 collection:control_collection shard:shard1
[junit4:junit4] 2> 9112 T37 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 9121 T37 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 9132 T37 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 9135 T37 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 9136 T37 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 9137 T37 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56743/collection1/
[junit4:junit4] 2> 9137 T37 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 9138 T37 oasc.SyncStrategy.syncToMe http://127.0.0.1:56743/collection1/ has no replicas
[junit4:junit4] 2> 9138 T37 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56743/collection1/
[junit4:junit4] 2> 9139 T37 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 9373 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9385 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 9398 T37 oasc.ZkController.register We are http://127.0.0.1:56743/collection1/ and leader is http://127.0.0.1:56743/collection1/
[junit4:junit4] 2> 9399 T37 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56743
[junit4:junit4] 2> 9400 T37 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 9400 T37 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 9400 T37 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 9403 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9406 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 9407 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 9422 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 9428 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 9430 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9432 T41 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ac39d name:ZooKeeperConnection Watcher:127.0.0.1:48085/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9433 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9435 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9440 T10 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 9534 T10 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 9538 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33198
[junit4:junit4] 2> 9539 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 9540 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 9540 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618
[junit4:junit4] 2> 9541 T10 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/solr.xml
[junit4:junit4] 2> 9541 T10 oasc.CoreContainer.<init> New CoreContainer 10511367
[junit4:junit4] 2> 9542 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/'
[junit4:junit4] 2> 9542 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/'
[junit4:junit4] 2> 9640 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 9641 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 9642 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 9643 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 9643 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 9644 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 9645 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 9646 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 9647 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 9648 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 9654 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 9655 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48085/solr
[junit4:junit4] 2> 9656 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 9658 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9660 T52 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ad5f07 name:ZooKeeperConnection Watcher:127.0.0.1:48085 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9661 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9664 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 9670 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9672 T54 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f49c95 name:ZooKeeperConnection Watcher:127.0.0.1:48085/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9673 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9679 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 10684 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33198_
[junit4:junit4] 2> 10705 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33198_
[junit4:junit4] 2> 10708 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 10709 T41 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 10709 T54 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 10712 T34 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 10717 T55 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 10718 T55 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10893 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10896 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:56743__collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:56743_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56743",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 10902 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:33198_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33198",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 10902 T35 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 10902 T35 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 10960 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10960 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10960 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 11720 T55 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/collection1
[junit4:junit4] 2> 11720 T55 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 11721 T55 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 11722 T55 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 11723 T55 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/collection1/'
[junit4:junit4] 2> 11724 T55 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/collection1/lib/README' to classloader
[junit4:junit4] 2> 11725 T55 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 11793 T55 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 11862 T55 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 11865 T55 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 11918 T55 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 12449 T55 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 12466 T55 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 12470 T55 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 12499 T55 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 12504 T55 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 12509 T55 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 12511 T55 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 12512 T55 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 12512 T55 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 12514 T55 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 12515 T55 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 12515 T55 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 12516 T55 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371252846618/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty1/
[junit4:junit4] 2> 12517 T55 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@614cee
[junit4:junit4] 2> 12519 T55 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 12519 T55 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty1
[junit4:junit4] 2> 12520 T55 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty1/index/
[junit4:junit4] 2> 12521 T55 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 12521 T55 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty1/index
[junit4:junit4] 2> 12524 T55 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a74fbf lockFactory=org.apache.lucene.store.NativeFSLockFactory@1792b88),segFN=segments_1,generation=1}
[junit4:junit4] 2> 12524 T55 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 12527 T55 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 12528 T55 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 12528 T55 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 12530 T55 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 12532 T55 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 12532 T55 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 12533 T55 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 12535 T55 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 12535 T55 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 12537 T55 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 12540 T55 oass.SolrIndexSearcher.<init> Opening Searcher@ad2d2d main
[junit4:junit4] 2> 12541 T55 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty1/tlog
[junit4:junit4] 2> 12542 T55 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 12542 T55 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 12562 T56 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ad2d2d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 12564 T55 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 12565 T55 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33198 collection:collection1 shard:shard1
[junit4:junit4] 2> 12566 T55 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 12577 T55 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 12580 T55 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 12580 T55 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 12581 T55 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33198/collection1/
[junit4:junit4] 2> 12581 T55 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 12582 T55 oasc.SyncStrategy.syncToMe http://127.0.0.1:33198/collection1/ has no replicas
[junit4:junit4] 2> 12582 T55 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33198/collection1/
[junit4:junit4] 2> 12582 T55 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 13972 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14004 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 14005 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 14004 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 14046 T55 oasc.ZkController.register We are http://127.0.0.1:33198/collection1/ and leader is http://127.0.0.1:33198/collection1/
[junit4:junit4] 2> 14046 T55 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33198
[junit4:junit4] 2> 14047 T55 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 14047 T55 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 14047 T55 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 14050 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14051 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 14052 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 14053 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14140 T10 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 14147 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48148
[junit4:junit4] 2> 14148 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 14149 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 14149 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237
[junit4:junit4] 2> 14150 T10 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/solr.xml
[junit4:junit4] 2> 14150 T10 oasc.CoreContainer.<init> New CoreContainer 30419203
[junit4:junit4] 2> 14151 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/'
[junit4:junit4] 2> 14151 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/'
[junit4:junit4] 2> 14238 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 14239 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 14239 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 14240 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 14241 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 14241 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 14242 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 14243 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 14244 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 14245 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 14251 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 14280 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48085/solr
[junit4:junit4] 2> 14280 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 14282 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 14290 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46ac4a name:ZooKeeperConnection Watcher:127.0.0.1:48085 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 14291 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 14294 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 14305 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 14316 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b184df name:ZooKeeperConnection Watcher:127.0.0.1:48085/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 14316 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 14322 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 15327 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48148_
[junit4:junit4] 2> 15329 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48148_
[junit4:junit4] 2> 15332 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15332 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15334 T54 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15334 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15334 T70 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15337 T34 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15343 T41 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15343 T71 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15345 T71 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15525 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15528 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33198__collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:33198_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33198",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 15534 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:48148_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48148",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 15535 T35 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 15535 T35 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 15541 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15542 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15542 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15541 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16347 T71 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/collection1
[junit4:junit4] 2> 16347 T71 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 16349 T71 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 16349 T71 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 16351 T71 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/collection1/'
[junit4:junit4] 2> 16352 T71 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/collection1/lib/README' to classloader
[junit4:junit4] 2> 16353 T71 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 16399 T71 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 16454 T71 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 16456 T71 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 16498 T71 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 16960 T71 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 16972 T71 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 16975 T71 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 17005 T71 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17011 T71 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17016 T71 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17018 T71 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 17018 T71 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 17019 T71 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17020 T71 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 17021 T71 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 17021 T71 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 17021 T71 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371252851237/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/
[junit4:junit4] 2> 17022 T71 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@614cee
[junit4:junit4] 2> 17022 T71 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 17023 T71 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2
[junit4:junit4] 2> 17023 T71 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index/
[junit4:junit4] 2> 17024 T71 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 17024 T71 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index
[junit4:junit4] 2> 17026 T71 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_1,generation=1}
[junit4:junit4] 2> 17026 T71 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 17029 T71 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 17030 T71 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 17030 T71 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 17032 T71 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 17033 T71 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 17033 T71 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 17034 T71 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 17035 T71 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 17036 T71 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 17038 T71 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 17041 T71 oass.SolrIndexSearcher.<init> Opening Searcher@e90b48 main
[junit4:junit4] 2> 17041 T71 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/tlog
[junit4:junit4] 2> 17042 T71 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 17042 T71 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 17058 T72 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e90b48 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 17065 T71 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 17066 T71 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48148 collection:collection1 shard:shard1
[junit4:junit4] 2> 17122 T71 oasc.ZkController.register We are http://127.0.0.1:48148/collection1/ and leader is http://127.0.0.1:33198/collection1/
[junit4:junit4] 2> 17123 T71 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48148
[junit4:junit4] 2> 17123 T71 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 17123 T71 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 17127 T71 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17129 T10 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 17129 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> ASYNC NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@d0a15d url=http://127.0.0.1:48148/collection1 node=127.0.0.1:48148_ C0_STATE=coll:collection1 core:collection1 props:{collection=collection1, node_name=127.0.0.1:48148_, base_url=http://127.0.0.1:48148, core=collection1, state=down}
[junit4:junit4] 2> 17130 T73 C0 P48148 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 17130 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 17131 T73 C0 P48148 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 17132 T73 C0 P48148 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 17132 T73 C0 P48148 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 17134 T73 C0 P48148 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 17136 T10 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17467 T50 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:48148__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C1_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 17619 T27 C1 P56743 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368ebd lockFactory=org.apache.lucene.store.NativeFSLockFactory@16988ac),segFN=segments_1,generation=1}
[junit4:junit4] 2> 17620 T27 C1 P56743 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 17700 T27 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1437862833469521920)]} 0 215
[junit4:junit4] 2> 17700 T28 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10001 (1437862833469521921)]} 0 215
[junit4:junit4] 2> ASYNC NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C2_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 17751 T48 C2 P33198 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a74fbf lockFactory=org.apache.lucene.store.NativeFSLockFactory@1792b88),segFN=segments_1,generation=1}
[junit4:junit4] 2> 17752 T48 C2 P33198 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 17757 T48 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1437862833607933952)]} 0 8
[junit4:junit4] 2> 17760 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10001 (1437862833610031104)]} 0 9
[junit4:junit4] 2> 17857 T28 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2 (1437862833716985856)]} 0 4
[junit4:junit4] 2> 17860 T27 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10002 (1437862833722228736)]} 0 2
[junit4:junit4] 2> 17865 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2 (1437862833725374464)]} 0 4
[junit4:junit4] 2> 17867 T48 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10002 (1437862833729568768)]} 0 2
[junit4:junit4] 2> 17910 T29 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[1 (-1437862833775706112)]} 0 1
[junit4:junit4] 2> 17910 T30 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10001 (-1437862833776754688)]} 0 0
[junit4:junit4] 2> 17916 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[1 (-1437862833783046144)]} 0 2
[junit4:junit4] 2> 17919 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10001 (-1437862833786191872)]} 0 1
[junit4:junit4] 2> 17925 T28 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3 (1437862833788289024)]} 0 4
[junit4:junit4] 2> 17927 T27 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10003 (1437862833791434752)]} 0 4
[junit4:junit4] 2> 17932 T48 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3 (1437862833796677632)]} 0 3
[junit4:junit4] 2> 17934 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10003 (1437862833798774784)]} 0 3
[junit4:junit4] 2> 18028 T29 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[3 (-1437862833900486656)]} 0 1
[junit4:junit4] 2> 18029 T30 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10003 (-1437862833901535232)]} 0 1
[junit4:junit4] 2> 18033 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[3 (-1437862833904680960)]} 0 1
[junit4:junit4] 2> 18036 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10003 (-1437862833907826688)]} 0 2
[junit4:junit4] 2> 18041 T28 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4 (1437862833910972416)]} 0 4
[junit4:junit4] 2> 18044 T27 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10004 (1437862833914118144)]} 0 4
[junit4:junit4] 2> 18049 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4 (1437862833919361024)]} 0 4
[junit4:junit4] 2> 18051 T48 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10004 (1437862833921458176)]} 0 3
[junit4:junit4] 2> 18094 T29 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10005 (1437862833965498369)]} 0 5
[junit4:junit4] 2> 18096 T30 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5 (1437862833965498368)]} 0 7
[junit4:junit4] 2> 18102 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10005 (1437862833974935552)]} 0 4
[junit4:junit4] 2> 18110 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5 (1437862833983324160)]} 0 8
[junit4:junit4] 2> 18124 T28 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10006 (1437862833998004224)]} 0 4
[junit4:junit4] 2> 18131 T48 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10006 (1437862834005344256)]} 0 3
[junit4:junit4] 2> 18132 T27 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6 (1437862834006392832)]} 0 3
[junit4:junit4] 2> 18138 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6 (1437862834013732864)]} 0 3
[junit4:junit4] 2> 18141 T30 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10007 (1437862834016878592)]} 0 3
[junit4:junit4] 2> 18147 T28 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1437862834023170048)]} 0 2
[junit4:junit4] 2> 18147 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10007 (1437862834022121472)]} 0 3
[junit4:junit4] 2> 18153 T49 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1437862834029461504)]} 0 2
[junit4:junit4] 2> 18218 T29 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10006 (-1437862834099716096)]} 0 1
[junit4:junit4] 2> 18222 T47 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10006 (-1437862834103910400)]} 0 1
[junit4:junit4] 2> 18223 T27 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[6 (-1437862834104958976)]} 0 0
[junit4:junit4] 2> 18227 T48 C2 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[6 (-1437862834108104704)]} 0 1
[junit4:junit4] 2> 18228 T30 C1 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10008 (1437862834107056128)]} 0 3
[junit4:junit4] 2> ASYNC NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C3_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 18241 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8 (1437862834112299008)]} 0 11
[junit4:junit4] 2> ASYNC NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C4_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 18241 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10008 (1437862834121736192)]} 0 10
[junit4:junit4] 2> 18247 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8 (1437862834126979072)]} 0 3
[junit4:junit4] 2> 18290 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10009 (1437862834173116416)]} 0 2
[junit4:junit4] 2> 18296 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[9 (1437862834179407872)]} 0 2
[junit4:junit4] 2> 18297 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10009 (1437862834179407872)]} 0 3
[junit4:junit4] 2> 18306 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10010 (1437862834188845056)]} 0 4
[junit4:junit4] 2> 18306 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[9 (1437862834185699328)]} 0 6
[junit4:junit4] 2> 18313 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10010 (1437862834196185088)]} 0 3
[junit4:junit4] 2> 18327 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10 (1437862834210865152)]} 0 4
[junit4:junit4] 2> 18334 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10 (1437862834218205184)]} 0 4
[junit4:junit4] 2> 18362 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10007 (-1437862834250711040)]} 0 0
[junit4:junit4] 2> 18366 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10007 (-1437862834254905344)]} 0 0
[junit4:junit4] 2> 18372 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10011 (1437862834259099648)]} 0 3
[junit4:junit4] 2> 18377 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10011 (1437862834264342528)]} 0 2
[junit4:junit4] 2> 18382 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[7 (-1437862834271682560)]} 0 1
[junit4:junit4] 2> 18386 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[7 (-1437862834275876864)]} 0 0
[junit4:junit4] 2> 18393 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[11 (1437862834280071168)]} 0 3
[junit4:junit4] 2> 18401 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[11 (1437862834288459776)]} 0 4
[junit4:junit4] 2> 18426 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10012 (1437862834313625600)]} 0 4
[junit4:junit4] 2> 18433 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10012 (1437862834322014208)]} 0 4
[junit4:junit4] 2> 18452 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[12 (1437862834341937152)]} 0 3
[junit4:junit4] 2> 18459 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[12 (1437862834349277184)]} 0 3
[junit4:junit4] 2> 18473 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10013 (1437862834365005824)]} 0 3
[junit4:junit4] 2> 18479 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10013 (1437862834371297280)]} 0 2
[junit4:junit4] 2> 18499 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[13 (1437862834391220224)]} 0 3
[junit4:junit4] 2> 18506 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[13 (1437862834398560256)]} 0 3
[junit4:junit4] 2> 18554 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18555 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48148__collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:48148_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48148",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 18562 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18562 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18563 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18564 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18580 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10009 (-1437862834479300608)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@d0a15d url=http://127.0.0.1:48148/collection1 node=127.0.0.1:48148_ C5_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:48148_, base_url=http://127.0.0.1:48148, core=collection1, state=recovering}
[junit4:junit4] 2> 18597 T63 C5 P48148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_1,generation=1}
[junit4:junit4] 2> 18598 T63 C5 P48148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 18599 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10009 (-1437862834484543488)]} 0 3
[junit4:junit4] 2> 18600 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10009 (-1437862834484543488)]} 0 17
[junit4:junit4] 2> 18605 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10014 (1437862834503417856)]} 0 2
[junit4:junit4] 2> 18607 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[9 (-1437862834507612160)]} 0 0
[junit4:junit4] 2> 18619 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[9 (-1437862834510757888)]} 0 4
[junit4:junit4] 2> 18620 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10014 (1437862834509709312)]} 0 4
[junit4:junit4] 2> 18621 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[9 (-1437862834510757888)]} 0 11
[junit4:junit4] 2> 18621 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10014 (1437862834509709312)]} 0 13
[junit4:junit4] 2> 18627 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[14 (1437862834526486528)]} 0 3
[junit4:junit4] 2> 18638 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[14 (1437862834532777984)]} 0 3
[junit4:junit4] 2> 18639 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[14 (1437862834532777984)]} 0 8
[junit4:junit4] 2> 18693 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10015 (1437862834594643968)]} 0 3
[junit4:junit4] 2> 18706 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10015 (1437862834601984000)]} 0 3
[junit4:junit4] 2> 18707 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10015 (1437862834601984000)]} 0 10
[junit4:junit4] 2> 18709 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[15 (1437862834612469760)]} 0 2
[junit4:junit4] 2> 18722 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[15 (1437862834618761216)]} 0 3
[junit4:junit4] 2> 18724 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[15 (1437862834618761216)]} 0 11
[junit4:junit4] 2> 18732 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10010 (-1437862834638684160)]} 0 1
[junit4:junit4] 2> 18739 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10010 (-1437862834642878464)]} 0 0
[junit4:junit4] 2> 18740 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10010 (-1437862834642878464)]} 0 5
[junit4:junit4] 2> 18747 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10016 (1437862834651267072)]} 0 3
[junit4:junit4] 2> 18750 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10 (-1437862834656509952)]} 0 1
[junit4:junit4] 2> 18757 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10 (-1437862834660704256)]} 0 1
[junit4:junit4] 2> 18762 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10016 (1437862834659655680)]} 0 3
[junit4:junit4] 2> 18763 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10 (-1437862834660704256)]} 0 10
[junit4:junit4] 2> 18764 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10016 (1437862834659655680)]} 0 12
[junit4:junit4] 2> 18771 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[16 (1437862834675384320)]} 0 4
[junit4:junit4] 2> 18784 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[16 (1437862834683772928)]} 0 3
[junit4:junit4] 2> 18785 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[16 (1437862834683772928)]} 0 10
[junit4:junit4] 2> 18858 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10012 (-1437862834770804736)]} 0 0
[junit4:junit4] 2> 18866 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10012 (-1437862834774999040)]} 0 1
[junit4:junit4] 2> 18867 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10012 (-1437862834774999040)]} 0 4
[junit4:junit4] 2> 18872 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10017 (1437862834783387648)]} 0 3
[junit4:junit4] 2> 18879 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[12 (-1437862834792824832)]} 0 0
[junit4:junit4] 2> 18882 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10017 (1437862834788630528)]} 0 3
[junit4:junit4] 2> 18883 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10017 (1437862834788630528)]} 0 8
[junit4:junit4] 2> 18886 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[12 (-1437862834797019136)]} 0 1
[junit4:junit4] 2> 18887 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[12 (-1437862834797019136)]} 0 4
[junit4:junit4] 2> 18893 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[17 (1437862834804359168)]} 0 3
[junit4:junit4] 2> 18895 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10014 (-1437862834809602048)]} 0 0
[junit4:junit4] 2> 18904 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10014 (-1437862834813796352)]} 0 1
[junit4:junit4] 2> 18904 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[17 (1437862834811699200)]} 0 2
[junit4:junit4] 2> 18905 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[17 (1437862834811699200)]} 0 9
[junit4:junit4] 2> 18905 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10014 (-1437862834813796352)]} 0 7
[junit4:junit4] 2> 18912 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10018 (1437862834824282112)]} 0 3
[junit4:junit4] 2> 18918 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[14 (-1437862834833719296)]} 0 1
[junit4:junit4] 2> 18922 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10018 (1437862834830573568)]} 0 3
[junit4:junit4] 2> 18924 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10018 (1437862834830573568)]} 0 9
[junit4:junit4] 2> 18925 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[14 (-1437862834837913600)]} 0 1
[junit4:junit4] 2> 18926 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[14 (-1437862834837913600)]} 0 5
[junit4:junit4] 2> 18932 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[18 (1437862834845253632)]} 0 3
[junit4:junit4] 2> 18942 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[18 (1437862834852593664)]} 0 2
[junit4:junit4] 2> 18943 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[18 (1437862834852593664)]} 0 7
[junit4:junit4] 2> 18971 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10015 (-1437862834889293824)]} 0 1
[junit4:junit4] 2> 18977 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10015 (-1437862834892439552)]} 0 0
[junit4:junit4] 2> 18979 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10015 (-1437862834892439552)]} 0 5
[junit4:junit4] 2> 18985 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10019 (1437862834901876736)]} 0 3
[junit4:junit4] 2> 18990 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[15 (-1437862834909216768)]} 0 0
[junit4:junit4] 2> 18996 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[15 (-1437862834912362496)]} 0 0
[junit4:junit4] 2> 18997 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[15 (-1437862834912362496)]} 0 4
[junit4:junit4] 2> 18998 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10019 (1437862834910265344)]} 0 2
[junit4:junit4] 2> 18999 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10019 (1437862834910265344)]} 0 8
[junit4:junit4] 2> 19004 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[19 (1437862834920751104)]} 0 4
[junit4:junit4] 2> 19014 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[19 (1437862834928091136)]} 0 2
[junit4:junit4] 2> 19015 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[19 (1437862834928091136)]} 0 8
[junit4:junit4] 2> 19098 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10020 (1437862835020365824)]} 0 2
[junit4:junit4] 2> 19108 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10020 (1437862835025608704)]} 0 2
[junit4:junit4] 2> 19109 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10020 (1437862835025608704)]} 0 8
[junit4:junit4] 2> 19113 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[20 (1437862835037143040)]} 0 2
[junit4:junit4] 2> 19124 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[20 (1437862835042385920)]} 0 3
[junit4:junit4] 2> 19125 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[20 (1437862835042385920)]} 0 9
[junit4:junit4] 2> 19140 T10 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 48148
[junit4:junit4] 2> 19141 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=30419203
[junit4:junit4] 2> 19144 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10017 (-1437862835070697472)]} 0 0
[junit4:junit4] 2> 19150 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10017 (-1437862835073843200)]} 0 1
[junit4:junit4] 2> 19151 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10017 (-1437862835073843200)]} 0 4
[junit4:junit4] 2> 19156 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10021 (1437862835081183232)]} 0 3
[junit4:junit4] 2> 19161 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[17 (-1437862835088523264)]} 0 1
[junit4:junit4] 2> 19167 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[17 (-1437862835091668992)]} 0 0
[junit4:junit4] 2> 19168 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10021 (1437862835088523264)]} 0 3
[junit4:junit4] 2> 19169 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[17 (-1437862835091668992)]} 0 5
[junit4:junit4] 2> 19169 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10021 (1437862835088523264)]} 0 9
[junit4:junit4] 2> 19174 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[21 (1437862835100057600)]} 0 2
[junit4:junit4] 2> 19184 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[21 (1437862835106349056)]} 0 2
[junit4:junit4] 2> 19185 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[21 (1437862835106349056)]} 0 7
[junit4:junit4] 2> 19263 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10022 (1437862835192332288)]} 0 3
[junit4:junit4] 2> 19274 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10022 (1437862835199672320)]} 0 2
[junit4:junit4] 2> 19275 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10022 (1437862835199672320)]} 0 9
[junit4:junit4] 2> 19278 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[22 (1437862835209109504)]} 0 2
[junit4:junit4] 2> 19288 T66 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[22 (1437862835215400960)]} 0 2
[junit4:junit4] 2> 19289 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[22 (1437862835215400960)]} 0 8
[junit4:junit4] 2> 19298 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10023 (1437862835229032448)]} 0 3
[junit4:junit4] 2> 19309 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10023 (1437862835236372480)]} 0 3
[junit4:junit4] 2> 19311 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10023 (1437862835236372480)]} 0 10
[junit4:junit4] 2> 19312 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[23 (1437862835243712512)]} 0 3
[junit4:junit4] 2> 19324 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[23 (1437862835250003968)]} 0 3
[junit4:junit4] 2> 19325 T47 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[23 (1437862835250003968)]} 0 10
[junit4:junit4] 2> 19392 T27 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10024 (1437862835328647168)]} 0 3
[junit4:junit4] 2> 19404 T63 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10024 (1437862835334938624)]} 0 3
[junit4:junit4] 2> 19405 T48 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10024 (1437862835334938624)]} 0 9
[junit4:junit4] 2> 19406 T30 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[24 (1437862835343327232)]} 0 3
[junit4:junit4] 2> 19417 T64 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[24 (1437862835349618688)]} 0 3
[junit4:junit4] 2> 19418 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[24 (1437862835349618688)]} 0 9
[junit4:junit4] 2> 19470 T50 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:48148__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 19471 T50 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&coreNodeName=127.0.0.1:48148__collection1&version=2&onlyIfLeader=true&nodeName=127.0.0.1:48148_&core=collection1&state=recovering&checkLive=true&wt=javabin} status=0 QTime=2012
[junit4:junit4] 2> 19476 T28 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10025 (1437862835417776128)]} 0 2
[junit4:junit4] 2> 19485 T65 C5 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10025 (1437862835423019008)]} 0 1
[junit4:junit4] 2> 19486 T49 C4 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10025 (1437862835423019008)]} 0 7
[junit4:junit4] 2> 19489 T29 C3 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[25 (1437862835430359040)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@d0a15d url=http://127.0.0.1:48148/collection1 node=127.0.0.1:48148_ C6_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:48148_, base_url=http://127.0.0.1:48148, core=collection1, state=recovering}
[junit4:junit4] 2> 19508 T66 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[25 (1437862835443990528)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C7_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 19510 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[25 (1437862835443990528)]} 0 10
[junit4:junit4] 2> ASYNC NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C8_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 19585 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10019 (-1437862835533119488)]} 0 0
[junit4:junit4] 2> 19592 T64 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10019 (-1437862835537313792)]} 0 1
[junit4:junit4] 2> 19593 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10019 (-1437862835537313792)]} 0 5
[junit4:junit4] 2> 19598 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10026 (1437862835544653824)]} 0 2
[junit4:junit4] 2> 19608 T65 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10026 (1437862835549896704)]} 0 2
[junit4:junit4] 2> 19609 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[19 (-1437862835557236736)]} 0 1
[junit4:junit4] 2> 19609 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10026 (1437862835549896704)]} 0 8
[junit4:junit4] 2> 19615 T66 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[19 (-1437862835561431040)]} 0 1
[junit4:junit4] 2> 19616 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[19 (-1437862835561431040)]} 0 4
[junit4:junit4] 2> 19621 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[26 (1437862835569819648)]} 0 2
[junit4:junit4] 2> 19631 T63 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[26 (1437862835575062528)]} 0 2
[junit4:junit4] 2> 19632 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[26 (1437862835575062528)]} 0 8
[junit4:junit4] 2> 19699 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10027 (1437862835650560000)]} 0 3
[junit4:junit4] 2> 19709 T64 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10027 (1437862835656851456)]} 0 2
[junit4:junit4] 2> 19710 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10027 (1437862835656851456)]} 0 7
[junit4:junit4] 2> 19721 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[27 (1437862835674677248)]} 0 2
[junit4:junit4] 2> 19731 T65 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[27 (1437862835679920128)]} 0 2
[junit4:junit4] 2> 19732 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[27 (1437862835679920128)]} 0 8
[junit4:junit4] 2> 19795 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10028 (1437862835751223296)]} 0 2
[junit4:junit4] 2> 19804 T66 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10028 (1437862835756466176)]} 0 2
[junit4:junit4] 2> 19805 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10028 (1437862835756466176)]} 0 7
[junit4:junit4] 2> 19818 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[28 (1437862835775340544)]} 0 2
[junit4:junit4] 2> 19829 T64 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[28 (1437862835781632000)]} 0 2
[junit4:junit4] 2> 19830 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[28 (1437862835781632000)]} 0 8
[junit4:junit4] 2> 19869 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10029 (1437862835827769344)]} 0 3
[junit4:junit4] 2> 19880 T65 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10029 (1437862835835109376)]} 0 3
[junit4:junit4] 2> 19881 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10029 (1437862835835109376)]} 0 9
[junit4:junit4] 2> 19893 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[29 (1437862835853983744)]} 0 3
[junit4:junit4] 2> 19903 T63 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[29 (1437862835860275200)]} 0 2
[junit4:junit4] 2> 19905 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[29 (1437862835860275200)]} 0 9
[junit4:junit4] 2> 19967 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10030 (1437862835931578368)]} 0 2
[junit4:junit4] 2> 19978 T64 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10030 (1437862835937869824)]} 0 2
[junit4:junit4] 2> 19980 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10030 (1437862835937869824)]} 0 9
[junit4:junit4] 2> 19990 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[30 (1437862835955695616)]} 0 2
[junit4:junit4] 2> 19999 T65 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[30 (1437862835961987072)]} 0 2
[junit4:junit4] 2> 20000 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[30 (1437862835961987072)]} 0 7
[junit4:junit4] 2> 20012 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10020 (-1437862835980861440)]} 0 0
[junit4:junit4] 2> 20018 T63 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10020 (-1437862835984007168)]} 0 1
[junit4:junit4] 2> 20019 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10020 (-1437862835984007168)]} 0 4
[junit4:junit4] 2> 20023 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10031 (1437862835991347200)]} 0 2
[junit4:junit4] 2> 20032 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[20 (-1437862836001832960)]} 0 1
[junit4:junit4] 2> 20032 T66 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10031 (1437862835995541504)]} 0 2
[junit4:junit4] 2> 20033 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10031 (1437862835995541504)]} 0 7
[junit4:junit4] 2> 20037 T64 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[20 (-1437862836004978688)]} 0 0
[junit4:junit4] 2> 20038 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[20 (-1437862836004978688)]} 0 3
[junit4:junit4] 2> 20043 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[31 (1437862836011270144)]} 0 2
[junit4:junit4] 2> 20052 T65 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[31 (1437862836016513024)]} 0 2
[junit4:junit4] 2> 20052 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[31 (1437862836016513024)]} 0 6
[junit4:junit4] 2> 20057 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10021 (-1437862836028047360)]} 0 1
[junit4:junit4] 2> 20062 T63 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[10021 (-1437862836030144512)]} 0 0
[junit4:junit4] 2> 20063 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10021 (-1437862836030144512)]} 0 4
[junit4:junit4] 2> 20068 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10032 (1437862836037484544)]} 0 2
[junit4:junit4] 2> 20074 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20075 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48148__collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48148_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48148",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 20077 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[21 (-1437862836047970304)]} 0 1
[junit4:junit4] 2> 20078 T66 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[10032 (1437862836043776000)]} 0 2
[junit4:junit4] 2> 20081 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10032 (1437862836043776000)]} 0 10
[junit4:junit4] 2> 20085 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 20086 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 20086 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 20086 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 20087 T64 C6 P48148 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33198/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {delete=[21 (-1437862836055310336)]} 0 1
[junit4:junit4] 2> 20089 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[21 (-1437862836055310336)]} 0 7
[junit4:junit4] 2> 20095 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[32 (1437862836064747520)]} 0 3
[junit4:junit4] 2> 20100 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[32 (1437862836071038976)]} 0 2
[junit4:junit4] 2> 20116 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10033 (1437862836087816192)]} 0 3
[junit4:junit4] 2> 20121 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10033 (1437862836093059072)]} 0 2
[junit4:junit4] 2> 20135 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[33 (1437862836107739136)]} 0 2
[junit4:junit4] 2> 20141 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[33 (1437862836114030592)]} 0 2
[junit4:junit4] 2> 20161 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10022 (-1437862836137099264)]} 0 0
[junit4:junit4] 2> 20165 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10022 (-1437862836141293568)]} 0 1
[junit4:junit4] 2> 20170 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10034 (1437862836144439296)]} 0 2
[junit4:junit4] 2> 20176 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10034 (1437862836150730752)]} 0 2
[junit4:junit4] 2> 20181 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[22 (-1437862836158070784)]} 0 1
[junit4:junit4] 2> 20184 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[22 (-1437862836161216512)]} 0 0
[junit4:junit4] 2> 20190 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[34 (1437862836165410816)]} 0 3
[junit4:junit4] 2> 20195 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[34 (1437862836170653696)]} 0 2
[junit4:junit4] 2> 20223 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10023 (-1437862836202110976)]} 0 0
[junit4:junit4] 2> 20227 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10023 (-1437862836206305280)]} 0 0
[junit4:junit4] 2> 20233 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10035 (1437862836210499584)]} 0 2
[junit4:junit4] 2> 20239 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10035 (1437862836216791040)]} 0 3
[junit4:junit4] 2> 20242 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[23 (-1437862836222033920)]} 0 1
[junit4:junit4] 2> 20245 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[23 (-1437862836225179648)]} 0 0
[junit4:junit4] 2> 20251 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[35 (1437862836229373952)]} 0 3
[junit4:junit4] 2> 20257 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[35 (1437862836235665408)]} 0 3
[junit4:junit4] 2> 20327 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10036 (1437862836309065728)]} 0 3
[junit4:junit4] 2> 20331 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10036 (1437862836314308608)]} 0 1
[junit4:junit4] 2> 20340 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10037 (1437862836323745792)]} 0 2
[junit4:junit4] 2> 20343 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[36 (1437862836326891520)]} 0 1
[junit4:junit4] 2> 20345 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10037 (1437862836327940096)]} 0 2
[junit4:junit4] 2> 20348 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[36 (1437862836331085824)]} 0 2
[junit4:junit4] 2> 20358 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[37 (1437862836341571584)]} 0 2
[junit4:junit4] 2> 20364 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[37 (1437862836347863040)]} 0 3
[junit4:junit4] 2> 20382 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10027 (-1437862836368834560)]} 0 0
[junit4:junit4] 2> 20386 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10027 (-1437862836371980288)]} 0 0
[junit4:junit4] 2> 20391 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10038 (1437862836376174592)]} 0 3
[junit4:junit4] 2> 20396 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10038 (1437862836381417472)]} 0 2
[junit4:junit4] 2> 20402 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[27 (-1437862836388757504)]} 0 1
[junit4:junit4] 2> 20405 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[27 (-1437862836392951808)]} 0 0
[junit4:junit4] 2> 20410 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[38 (1437862836396097536)]} 0 2
[junit4:junit4] 2> 20414 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[38 (1437862836401340416)]} 0 1
[junit4:junit4] 2> 20447 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10039 (1437862836435943424)]} 0 2
[junit4:junit4] 2> 20453 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10039 (1437862836441186304)]} 0 2
[junit4:junit4] 2> 20466 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[39 (1437862836454817792)]} 0 2
[junit4:junit4] 2> 20472 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[39 (1437862836461109248)]} 0 2
[junit4:junit4] 2> 20544 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10040 (1437862836536606720)]} 0 2
[junit4:junit4] 2> 20548 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10040 (1437862836541849600)]} 0 1
[junit4:junit4] 2> 20564 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[40 (1437862836557578240)]} 0 2
[junit4:junit4] 2> 20569 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[40 (1437862836562821120)]} 0 2
[junit4:junit4] 2> 20603 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10041 (1437862836598472704)]} 0 3
[junit4:junit4] 2> 20608 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10041 (1437862836603715584)]} 0 2
[junit4:junit4] 2> 20623 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[41 (1437862836619444224)]} 0 2
[junit4:junit4] 2> 20627 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[41 (1437862836624687104)]} 0 2
[junit4:junit4] 2> 20666 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10042 (1437862836664532992)]} 0 3
[junit4:junit4] 2> 20671 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10042 (1437862836669775872)]} 0 2
[junit4:junit4] 2> 20684 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[42 (1437862836684455936)]} 0 2
[junit4:junit4] 2> 20690 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[42 (1437862836688650240)]} 0 3
[junit4:junit4] 2> 20693 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10043 (1437862836692844544)]} 0 2
[junit4:junit4] 2> 20698 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10043 (1437862836698087424)]} 0 2
[junit4:junit4] 2> 20711 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[43 (1437862836712767488)]} 0 1
[junit4:junit4] 2> 20716 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[43 (1437862836716961792)]} 0 2
[junit4:junit4] 2> 20755 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10044 (1437862836757856256)]} 0 2
[junit4:junit4] 2> 20761 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10044 (1437862836764147712)]} 0 3
[junit4:junit4] 2> 20773 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[44 (1437862836776730624)]} 0 2
[junit4:junit4] 2> 20778 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[44 (1437862836783022080)]} 0 2
[junit4:junit4] 2> 20833 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10045 (1437862836840693760)]} 0 2
[junit4:junit4] 2> 20838 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10045 (1437862836845936640)]} 0 1
[junit4:junit4] 2> 20850 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[45 (1437862836857470976)]} 0 2
[junit4:junit4] 2> 20855 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[45 (1437862836863762432)]} 0 2
[junit4:junit4] 2> 20893 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10046 (1437862836903608320)]} 0 1
[junit4:junit4] 2> 20898 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10046 (1437862836907802624)]} 0 2
[junit4:junit4] 2> 20909 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[46 (1437862836920385536)]} 0 1
[junit4:junit4] 2> 20913 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[46 (1437862836924579840)]} 0 1
[junit4:junit4] 2> 20984 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10028 (-1437862837000077312)]} 0 0
[junit4:junit4] 2> 20988 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10028 (-1437862837003223040)]} 0 1
[junit4:junit4] 2> 20992 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10047 (1437862837006368768)]} 0 2
[junit4:junit4] 2> 20996 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10047 (1437862837011611648)]} 0 1
[junit4:junit4] 2> 20999 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[28 (-1437862837015805952)]} 0 1
[junit4:junit4] 2> 21001 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[28 (-1437862837017903104)]} 0 0
[junit4:junit4] 2> 21005 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[47 (1437862837021048832)]} 0 1
[junit4:junit4] 2> 21009 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[47 (1437862837025243136)]} 0 1
[junit4:junit4] 2> 21011 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10031 (-1437862837028388864)]} 0 0
[junit4:junit4] 2> 21014 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10031 (-1437862837031534592)]} 0 0
[junit4:junit4] 2> 21019 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10048 (1437862837034680320)]} 0 2
[junit4:junit4] 2> 21023 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10048 (1437862837039923200)]} 0 1
[junit4:junit4] 2> 21024 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[31 (-1437862837042020352)]} 0 0
[junit4:junit4] 2> 21027 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[31 (-1437862837045166080)]} 0 1
[junit4:junit4] 2> 21031 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[48 (1437862837048311808)]} 0 2
[junit4:junit4] 2> 21035 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[48 (1437862837052506112)]} 0 2
[junit4:junit4] 2> 21068 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10049 (1437862837087109120)]} 0 2
[junit4:junit4] 2> 21073 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10049 (1437862837092352000)]} 0 2
[junit4:junit4] 2> 21079 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[49 (1437862837098643456)]} 0 2
[junit4:junit4] 2> 21084 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[49 (1437862837102837760)]} 0 2
[junit4:junit4] 2> 21144 T10 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 21145 T10 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 21147 T10 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=127.0.0.1:48148__collection1core=collection1
[junit4:junit4] 2> 21154 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10032 (-1437862837178335232)]} 0 0
[junit4:junit4] 2> 21157 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10032 (-1437862837181480960)]} 0 0
[junit4:junit4] 2> 21161 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10050 (1437862837184626688)]} 0 1
[junit4:junit4] 2> 21165 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[32 (-1437862837189869568)]} 0 0
[junit4:junit4] 2> 21165 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10050 (1437862837188820992)]} 0 1
[junit4:junit4] 2> 21169 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[32 (-1437862837193015296)]} 0 1
[junit4:junit4] 2> 21173 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[50 (1437862837197209600)]} 0 2
[junit4:junit4] 2> 21178 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[50 (1437862837201403904)]} 0 2
[junit4:junit4] 2> 21253 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10034 (-1437862837282144256)]} 0 0
[junit4:junit4] 2> 21256 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10034 (-1437862837285289984)]} 0 0
[junit4:junit4] 2> 21260 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10051 (1437862837288435712)]} 0 1
[junit4:junit4] 2> 21264 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10051 (1437862837292630016)]} 0 1
[junit4:junit4] 2> 21265 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[34 (-1437862837294727168)]} 0 1
[junit4:junit4] 2> 21268 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[34 (-1437862837296824320)]} 0 1
[junit4:junit4] 2> 21272 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[51 (1437862837301018624)]} 0 2
[junit4:junit4] 2> 21278 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[51 (1437862837306261504)]} 0 2
[junit4:junit4] 2> 21323 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10052 (1437862837354496000)]} 0 1
[junit4:junit4] 2> 21328 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10052 (1437862837358690304)]} 0 1
[junit4:junit4] 2> 21337 T27 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[52 (1437862837369176064)]} 0 1
[junit4:junit4] 2> 21341 T47 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[52 (1437862837373370368)]} 0 1
[junit4:junit4] 2> 21352 T28 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10053 (1437862837384904704)]} 0 2
[junit4:junit4] 2> 21356 T50 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10053 (1437862837389099008)]} 0 1
[junit4:junit4] 2> 21366 T29 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[53 (1437862837399584768)]} 0 2
[junit4:junit4] 2> 21371 T49 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[53 (1437862837404827648)]} 0 2
[junit4:junit4] 2> 21410 T30 C8 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10054 (1437862837445722112)]} 0 1
[junit4:junit4] 2> 21414 T48 C7 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10054 (1437862837449916416)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C9_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 21423 T28 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10035 (-1437862837460402177)]} 0 0
[junit4:junit4] 2> 21424 T27 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[54 (1437862837460402176)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C10_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 21426 T47 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10035 (-1437862837463547904)]} 0 0
[junit4:junit4] 2> 21428 T50 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[54 (1437862837464596480)]} 0 2
[junit4:junit4] 2> 21430 T29 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10055 (1437862837466693632)]} 0 1
[junit4:junit4] 2> 21433 T30 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[35 (-1437862837470887936)]} 0 0
[junit4:junit4] 2> 21435 T48 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10055 (1437862837470887936)]} 0 2
[junit4:junit4] 2> 21436 T49 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[35 (-1437862837474033664)]} 0 0
[junit4:junit4] 2> 21441 T28 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[55 (1437862837477179392)]} 0 2
[junit4:junit4] 2> 21445 T47 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[55 (1437862837482422272)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@d0a15d url=http://127.0.0.1:48148/collection1 node=127.0.0.1:48148_ C11_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:48148_, base_url=http://127.0.0.1:48148, core=collection1, state=down}
[junit4:junit4] 2> 21472 T73 C11 P48148 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33198/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 21476 T73 C11 P48148 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 21481 T73 C11 P48148 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48148 START replicas=[http://127.0.0.1:33198/collection1/] nUpdates=100
[junit4:junit4] 2> 21488 T73 C11 P48148 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 21489 T73 C11 P48148 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 21489 T73 C11 P48148 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 21489 T73 C11 P48148 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 21490 T73 C11 P48148 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 21490 T73 C11 P48148 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33198/collection1/. core=collection1
[junit4:junit4] 2> 21490 T73 C11 P48148 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 21494 T48 C10 P33198 oasc.SolrCore.execute [collection1] webapp= path=/get params={qt=/get&version=2&distrib=false&getVersions=100&wt=javabin} status=0 QTime=3
[junit4:junit4] 2> 21497 T49 C10 P33198 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 21527 T27 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10056 (1437862837566308352)]} 0 3
[junit4:junit4] 2> 21532 T47 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10056 (1437862837572599808)]} 0 2
[junit4:junit4] 2> 21535 T29 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[56 (1437862837576794112)]} 0 1
[junit4:junit4] 2> 21540 T50 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[56 (1437862837580988416)]} 0 2
[junit4:junit4] 2> 21552 T30 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10057 (1437862837594619904)]} 0 2
[junit4:junit4] 2> 21557 T48 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10057 (1437862837598814208)]} 0 2
[junit4:junit4] 2> 21559 T28 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[57 (1437862837601959936)]} 0 1
[junit4:junit4] 2> 21564 T50 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[57 (1437862837606154240)]} 0 2
[junit4:junit4] 2> 21593 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21594 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48148__collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:48148_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48148",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 21595 T29 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10058 (1437862837638660096)]} 0 2
[junit4:junit4] 2> 21601 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21601 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21601 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21601 T47 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10058 (1437862837644951552)]} 0 3
[junit4:junit4] 2> 21602 T30 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[58 (1437862837647048704)]} 0 2
[junit4:junit4] 2> 21602 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21608 T48 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[58 (1437862837653340160)]} 0 1
[junit4:junit4] 2> 21618 T27 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10059 (1437862837663825920)]} 0 2
[junit4:junit4] 2> 21622 T50 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10059 (1437862837668020224)]} 0 1
[junit4:junit4] 2> 21623 T28 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[59 (1437862837668020224)]} 0 2
[junit4:junit4] 2> 21627 T47 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[59 (1437862837673263104)]} 0 1
[junit4:junit4] 2> 21665 T49 C10 P33198 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a74fbf lockFactory=org.apache.lucene.store.NativeFSLockFactory@1792b88),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a74fbf lockFactory=org.apache.lucene.store.NativeFSLockFactory@1792b88),segFN=segments_2,generation=2}
[junit4:junit4] 2> 21666 T49 C10 P33198 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 21717 T29 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10060 (1437862837767634944)]} 0 1
[junit4:junit4] 2> 21722 T30 C9 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[60 (1437862837771829248)]} 0 2
[junit4:junit4] 2> 21722 T48 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10060 (1437862837772877824)]} 0 2
[junit4:junit4] 2> 21728 T50 C10 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[60 (1437862837778120704)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C12_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 21791 T28 C12 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10061 (1437862837844180992)]} 0 3
[junit4:junit4] 2> 21794 T29 C12 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[61 (1437862837848375296)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C13_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 21797 T47 C13 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10061 (1437862837850472448)]} 0 2
[junit4:junit4] 2> 21799 T48 C13 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[61 (1437862837853618176)]} 0 2
[junit4:junit4] 2> 21831 T49 C13 P33198 oass.SolrIndexSearcher.<init> Opening Searcher@e61aa2 realtime
[junit4:junit4] 2> 21831 T49 C13 P33198 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 21832 T49 C13 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={commit=true&openSearcher=false&softCommit=false&commit_end_point=true&version=2&waitSearcher=true&wt=javabin} {commit=} 0 335
[junit4:junit4] 2> ASYNC NEW_CORE C14 name=collection1 org.apache.solr.core.SolrCore@d0a15d url=http://127.0.0.1:48148/collection1 node=127.0.0.1:48148_ C14_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:48148_, base_url=http://127.0.0.1:48148, core=collection1, state=down}
[junit4:junit4] 2> 21851 T73 C14 P48148 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 21852 T73 C14 P48148 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 21857 T73 C14 P48148 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 21863 T27 C12 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10062 (1437862837919678464)]} 0 2
[junit4:junit4] 2> 21864 T30 C12 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[62 (1437862837921775616)]} 0 2
[junit4:junit4] 2> 21868 T49 C13 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10062 (1437862837924921344)]} 0 2
[junit4:junit4] 2> 21869 T50 C13 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[62 (1437862837925969920)]} 0 2
[junit4:junit4] 2> 21873 T48 C13 P33198 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 21874 T48 C13 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=15
[junit4:junit4] 2> 21875 T73 C14 P48148 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 21875 T73 C14 P48148 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 21875 T73 C14 P48148 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 21878 T47 C13 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&command=filelist&generation=2&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 21879 T73 C14 P48148 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 21880 T73 C14 P48148 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index.20130614193419058
[junit4:junit4] 2> 21881 T73 C14 P48148 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@17a44a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4780f1) fullCopy=false
[junit4:junit4] 2> 21886 T49 C13 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&command=filecontent&file=_0.cfs&generation=2&wt=filestream} status=0 QTime=1
[junit4:junit4] 2> 21893 T50 C13 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&command=filecontent&file=_0.si&generation=2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 21897 T48 C13 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&command=filecontent&file=_0_1.del&generation=2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C15 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C15_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 21908 T47 C15 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&command=filecontent&file=_0.cfe&generation=2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 21912 T49 C15 P33198 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&command=filecontent&file=segments_2&generation=2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C16 name=collection1 org.apache.solr.core.SolrCore@d0a15d url=http://127.0.0.1:48148/collection1 node=127.0.0.1:48148_ C16_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:48148_, base_url=http://127.0.0.1:48148, core=collection1, state=down}
[junit4:junit4] 2> 21913 T73 C16 P48148 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 21920 T73 C16 P48148 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 21920 T73 C16 P48148 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 21920 T73 C16 P48148 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 21937 T73 C16 P48148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_2,generation=2}
[junit4:junit4] 2> 21937 T73 C16 P48148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 21940 T73 C16 P48148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_2,generation=2}
[junit4:junit4] 2> 21941 T73 C16 P48148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 21941 T73 C16 P48148 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> ASYNC NEW_CORE C17 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C17_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 21945 T29 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[63 (1437862838006710273)]} 0 1
[junit4:junit4] 2> 21945 T28 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10063 (1437862838006710272)]} 0 2
[junit4:junit4] 2> 21947 T73 C16 P48148 oass.SolrIndexSearcher.<init> Opening Searcher@707b0c main
[junit4:junit4] 2> 21949 T50 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[63 (1437862838010904576)]} 0 1
[junit4:junit4] 2> 21950 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10063 (1437862838010904577)]} 0 2
[junit4:junit4] 2> 21951 T72 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@707b0c main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c37/12)}
[junit4:junit4] 2> 21953 T73 C16 P48148 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index.20130614193419058 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index.20130614193419058;done=true>>]
[junit4:junit4] 2> 21953 T73 C16 P48148 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index.20130614193419058
[junit4:junit4] 2> 21954 T73 C16 P48148 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index.20130614193419058
[junit4:junit4] 2> 21956 T73 C16 P48148 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 21956 T81 C16 P48148 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=5382
[junit4:junit4] 2> 21958 T81 C16 P48148 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 21959 T81 C16 P48148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d1b78a lockFactory=org.apache.lucene.store.NativeFSLockFactory@8310cb),segFN=segments_3,generation=3}
[junit4:junit4] 2> 21959 T81 C16 P48148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 21959 T81 C16 P48148 oass.SolrIndexSearcher.<init> Opening Searcher@ad2963 main
[junit4:junit4] 2> 21960 T81 C16 P48148 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 21962 T72 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ad2963 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c37/12)}
[junit4:junit4] 2> 21963 T81 C16 P48148 oasup.LogUpdateProcessor.finish [collection1] {} 0 7
[junit4:junit4] 2> 21963 T81 C16 P48148 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=5382}
[junit4:junit4] 2> 21963 T73 C16 P48148 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 21964 T73 C16 P48148 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 21964 T73 C16 P48148 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 21966 T73 C16 P48148 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 21966 T10 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@d0a15d
[junit4:junit4] 2> 21983 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=37,cumulative_deletesById=18,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 21984 T10 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 21984 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 21985 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 21986 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 21987 T10 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 21988 T10 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2;done=false>>]
[junit4:junit4] 2> 21988 T10 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2
[junit4:junit4] 2> 21988 T10 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index;done=false>>]
[junit4:junit4] 2> 21989 T10 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index
[junit4:junit4] 2> 21991 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21991 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21991 T70 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 21992 T41 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 21992 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 21993 T70 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 21993 T70 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 21993 T34 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 21994 T54 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 22014 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 22041 T27 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[64 (1437862838106324992)]} 0 2
[junit4:junit4] 2> 22041 T30 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10064 (1437862838106324993)]} 0 2
[junit4:junit4] 2> 22045 T50 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10064 (1437862838111567872)]} 0 1
[junit4:junit4] 2> 22046 T47 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[64 (1437862838111567873)]} 0 2
[junit4:junit4] 2> 22146 T28 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10065 (1437862838216425472)]} 0 2
[junit4:junit4] 2> 22146 T29 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[65 (1437862838217474048)]} 0 1
[junit4:junit4] 2> 22150 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10065 (1437862838221668352)]} 0 2
[junit4:junit4] 2> 22151 T49 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[65 (1437862838221668353)]} 0 2
[junit4:junit4] 2> 22168 T30 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10066 (1437862838239494144)]} 0 2
[junit4:junit4] 2> 22169 T27 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[66 (1437862838239494145)]} 0 2
[junit4:junit4] 2> 22172 T47 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[66 (1437862838244737024)]} 0 1
[junit4:junit4] 2> 22173 T50 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10066 (1437862838244737025)]} 0 2
[junit4:junit4] 2> 22260 T28 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[67 (1437862838337011712)]} 0 2
[junit4:junit4] 2> 22260 T29 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10067 (1437862838338060288)]} 0 1
[junit4:junit4] 2> 22265 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[67 (1437862838342254592)]} 0 1
[junit4:junit4] 2> 22266 T47 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10067 (1437862838342254593)]} 0 2
[junit4:junit4] 2> 22333 T27 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[36 (-1437862838414606336)]} 0 1
[junit4:junit4] 2> 22333 T30 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10036 (-1437862838414606337)]} 0 0
[junit4:junit4] 2> 22336 T50 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[36 (-1437862838417752064)]} 0 0
[junit4:junit4] 2> 22336 T49 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10036 (-1437862838417752065)]} 0 0
[junit4:junit4] 2> 22341 T28 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[68 (1437862838420897792)]} 0 1
[junit4:junit4] 2> 22341 T29 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10068 (1437862838421946368)]} 0 2
[junit4:junit4] 2> 22345 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[68 (1437862838426140672)]} 0 2
[junit4:junit4] 2> 22346 T47 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10068 (1437862838426140673)]} 0 2
[junit4:junit4] 2> 22366 T27 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[37 (-1437862838448160768)]} 0 0
[junit4:junit4] 2> 22366 T30 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10037 (-1437862838449209344)]} 0 0
[junit4:junit4] 2> 22369 T50 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[37 (-1437862838451306496)]} 0 1
[junit4:junit4] 2> 22369 T49 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10037 (-1437862838452355072)]} 0 0
[junit4:junit4] 2> 22373 T28 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[69 (1437862838455500800)]} 0 1
[junit4:junit4] 2> 22373 T29 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10069 (1437862838455500801)]} 0 1
[junit4:junit4] 2> 22378 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[69 (1437862838460743680)]} 0 2
[junit4:junit4] 2> 22379 T47 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10069 (1437862838460743681)]} 0 2
[junit4:junit4] 2> 22414 T27 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[70 (1437862838497443840)]} 0 2
[junit4:junit4] 2> 22415 T30 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10070 (1437862838497443841)]} 0 3
[junit4:junit4] 2> 22419 T50 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[70 (1437862838502686720)]} 0 2
[junit4:junit4] 2> 22422 T49 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10070 (1437862838505832448)]} 0 2
[junit4:junit4] 2> 22498 T28 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[40 (-1437862838587621376)]} 0 0
[junit4:junit4] 2> 22500 T27 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10040 (-1437862838589718528)]} 0 0
[junit4:junit4] 2> 22501 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[40 (-1437862838590767104)]} 0 0
[junit4:junit4] 2> 22503 T47 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10040 (-1437862838592864256)]} 0 0
[junit4:junit4] 2> 22505 T30 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[71 (1437862838593912832)]} 0 2
[junit4:junit4] 2> 22506 T29 C17 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10071 (1437862838596009984)]} 0 1
[junit4:junit4] 2> 22510 T49 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[71 (1437862838598107136)]} 0 2
[junit4:junit4] 2> 22510 T48 C15 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10071 (1437862838599155712)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C18 name=collection1 org.apache.solr.core.SolrCore@141e0b5 url=http://127.0.0.1:56743/collection1 node=127.0.0.1:56743_ C18_STATE=coll:control_collection core:collection1 props:{shard=shard1, collection=control_collection, node_name=127.0.0.1:56743_, base_url=http://127.0.0.1:56743, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 22599 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[72 (1437862838686187520)]} 0 7
[junit4:junit4] 2> 22599 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10072 (1437862838686187521)]} 0 7
[junit4:junit4] 2> ASYNC NEW_CORE C19 name=collection1 org.apache.solr.core.SolrCore@7e856 url=http://127.0.0.1:33198/collection1 node=127.0.0.1:33198_ C19_STATE=coll:collection1 core:collection1 props:{shard=shard1, collection=collection1, node_name=127.0.0.1:33198_, base_url=http://127.0.0.1:33198, core=collection1, state=active, leader=true}
[junit4:junit4] 2> 22603 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[72 (1437862838696673280)]} 0 1
[junit4:junit4] 2> 22603 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10072 (1437862838696673281)]} 0 1
[junit4:junit4] 2> 22623 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[73 (1437862838717644800)]} 0 1
[junit4:junit4] 2> 22624 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10073 (1437862838717644801)]} 0 2
[junit4:junit4] 2> 22627 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[73 (1437862838721839104)]} 0 1
[junit4:junit4] 2> 22628 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10073 (1437862838722887680)]} 0 1
[junit4:junit4] 2> 22675 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[42 (-1437862838773219328)]} 0 0
[junit4:junit4] 2> 22675 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10042 (-1437862838773219329)]} 0 0
[junit4:junit4] 2> 22678 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[42 (-1437862838776365056)]} 0 0
[junit4:junit4] 2> 22678 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10042 (-1437862838776365057)]} 0 0
[junit4:junit4] 2> 22682 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[74 (1437862838779510784)]} 0 2
[junit4:junit4] 2> 22683 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10074 (1437862838779510785)]} 0 2
[junit4:junit4] 2> 22686 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[74 (1437862838783705088)]} 0 2
[junit4:junit4] 2> 22687 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10074 (1437862838784753664)]} 0 2
[junit4:junit4] 2> 22736 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[45 (-1437862838837182464)]} 0 0
[junit4:junit4] 2> 22737 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10045 (-1437862838838231040)]} 0 1
[junit4:junit4] 2> 22739 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[45 (-1437862838840328192)]} 0 0
[junit4:junit4] 2> 22739 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10045 (-1437862838840328193)]} 0 0
[junit4:junit4] 2> 22743 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[75 (1437862838843473920)]} 0 1
[junit4:junit4] 2> 22744 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10075 (1437862838843473921)]} 0 2
[junit4:junit4] 2> 22747 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[75 (1437862838847668224)]} 0 1
[junit4:junit4] 2> 22748 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10075 (1437862838848716800)]} 0 1
[junit4:junit4] 2> 22764 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[76 (1437862838865494016)]} 0 1
[junit4:junit4] 2> 22766 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10076 (1437862838866542592)]} 0 2
[junit4:junit4] 2> 22768 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[76 (1437862838869688320)]} 0 1
[junit4:junit4] 2> 22770 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10076 (1437862838871785472)]} 0 1
[junit4:junit4] 2> 22777 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[46 (-1437862838880174080)]} 0 0
[junit4:junit4] 2> 22779 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10046 (-1437862838882271232)]} 0 0
[junit4:junit4] 2> 22780 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[46 (-1437862838882271232)]} 0 1
[junit4:junit4] 2> 22782 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10046 (-1437862838885416960)]} 0 0
[junit4:junit4] 2> 22784 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[77 (1437862838885416960)]} 0 2
[junit4:junit4] 2> 22786 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10077 (1437862838888562688)]} 0 1
[junit4:junit4] 2> 22788 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[77 (1437862838889611264)]} 0 2
[junit4:junit4] 2> 22791 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10077 (1437862838893805568)]} 0 2
[junit4:junit4] 2> 22818 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[78 (1437862838921068544)]} 0 2
[junit4:junit4] 2> 22821 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10078 (1437862838925262848)]} 0 2
[junit4:junit4] 2> 22824 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[78 (1437862838928408576)]} 0 2
[junit4:junit4] 2> 22826 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10078 (1437862838930505728)]} 0 1
[junit4:junit4] 2> 22880 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[79 (1437862838986080256)]} 0 2
[junit4:junit4] 2> 22881 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10079 (1437862838988177408)]} 0 2
[junit4:junit4] 2> 22884 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[79 (1437862838991323136)]} 0 1
[junit4:junit4] 2> 22885 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10079 (1437862838991323137)]} 0 2
[junit4:junit4] 2> 22944 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[80 (1437862839054237696)]} 0 1
[junit4:junit4] 2> 22945 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10080 (1437862839055286272)]} 0 1
[junit4:junit4] 2> 22948 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[80 (1437862839058432000)]} 0 2
[junit4:junit4] 2> 22949 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10080 (1437862839059480576)]} 0 1
[junit4:junit4] 2> 22984 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[49 (-1437862839097229312)]} 0 1
[junit4:junit4] 2> 22984 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10049 (-1437862839097229313)]} 0 0
[junit4:junit4] 2> 22987 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[49 (-1437862839100375040)]} 0 1
[junit4:junit4] 2> 22987 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10049 (-1437862839100375041)]} 0 0
[junit4:junit4] 2> 22991 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[81 (1437862839103520768)]} 0 2
[junit4:junit4] 2> 22991 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10081 (1437862839103520769)]} 0 1
[junit4:junit4] 2> 22994 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[81 (1437862839107715072)]} 0 1
[junit4:junit4] 2> 22995 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10081 (1437862839107715073)]} 0 1
[junit4:junit4] 2> 23090 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[56 (-1437862839208378368)]} 0 1
[junit4:junit4] 2> 23090 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10056 (-1437862839208378369)]} 0 0
[junit4:junit4] 2> 23093 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[56 (-1437862839210475520)]} 0 1
[junit4:junit4] 2> 23093 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10056 (-1437862839211524096)]} 0 0
[junit4:junit4] 2> 23096 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[82 (1437862839213621248)]} 0 1
[junit4:junit4] 2> 23097 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10082 (1437862839214669824)]} 0 1
[junit4:junit4] 2> 23101 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[82 (1437862839217815552)]} 0 2
[junit4:junit4] 2> 23103 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10082 (1437862839220961280)]} 0 1
[junit4:junit4] 2> 23109 T35 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23110 T35 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48148__collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:48148_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48148",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 23117 T34 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 23117 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 23117 T41 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 23123 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[57 (-1437862839242981376)]} 0 0
[junit4:junit4] 2> 23125 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10057 (-1437862839245078528)]} 0 0
[junit4:junit4] 2> 23126 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[57 (-1437862839246127104)]} 0 0
[junit4:junit4] 2> 23128 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10057 (-1437862839248224256)]} 0 0
[junit4:junit4] 2> 23133 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[83 (1437862839248224256)]} 0 5
[junit4:junit4] 2> 23133 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10083 (1437862839251369984)]} 0 3
[junit4:junit4] 2> 23137 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[83 (1437862839256612864)]} 0 2
[junit4:junit4] 2> 23137 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10083 (1437862839256612865)]} 0 1
[junit4:junit4] 2> 23208 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[58 (-1437862839332110336)]} 0 0
[junit4:junit4] 2> 23208 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10058 (-1437862839332110337)]} 0 0
[junit4:junit4] 2> 23211 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[58 (-1437862839335256064)]} 0 1
[junit4:junit4] 2> 23211 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10058 (-1437862839335256065)]} 0 0
[junit4:junit4] 2> 23214 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[84 (1437862839337353216)]} 0 1
[junit4:junit4] 2> 23215 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10084 (1437862839338401792)]} 0 2
[junit4:junit4] 2> 23219 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[84 (1437862839341547520)]} 0 2
[junit4:junit4] 2> 23220 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10084 (1437862839343644672)]} 0 2
[junit4:junit4] 2> 23239 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[59 (-1437862839364616192)]} 0 0
[junit4:junit4] 2> 23240 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10059 (-1437862839365664768)]} 0 0
[junit4:junit4] 2> 23243 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[59 (-1437862839367761920)]} 0 1
[junit4:junit4] 2> 23243 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[10059 (-1437862839368810496)]} 0 0
[junit4:junit4] 2> 23247 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[85 (1437862839371956224)]} 0 1
[junit4:junit4] 2> 23248 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10085 (1437862839371956225)]} 0 2
[junit4:junit4] 2> 23250 T50 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[85 (1437862839376150528)]} 0 1
[junit4:junit4] 2> 23252 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10085 (1437862839377199104)]} 0 1
[junit4:junit4] 2> 23343 T27 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[86 (1437862839472619520)]} 0 2
[junit4:junit4] 2> 23344 T28 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10086 (1437862839473668096)]} 0 1
[junit4:junit4] 2> 23346 T49 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[86 (1437862839476813824)]} 0 1
[junit4:junit4] 2> 23347 T48 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10086 (1437862839477862400)]} 0 1
[junit4:junit4] 2> 23384 T30 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[87 (1437862839515611136)]} 0 2
[junit4:junit4] 2> 23384 T29 C18 P56743 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[10087 (1437862839515611137)]} 0 1
[junit4:junit4] 2> 23387 T47 C19 P33198 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[87 (1437862839518756864)]} 0 1
[junit4:junit4] 2> 23388 T49 C19
[...truncated too long message...]
it4:junit4] 2> 121989 T107 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 121992 T94 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 121993 T106 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 121994 T94 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 121995 T106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 121996 T106 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33198__collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33198_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33198",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 121997 T94 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 121997 T94 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 121997 T94 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 121997 T94 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48148/collection1/
[junit4:junit4] 2> 121998 T94 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 121998 T94 oasc.SyncStrategy.syncToMe http://127.0.0.1:48148/collection1/ has no replicas
[junit4:junit4] 2> 121998 T94 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48148/collection1/
[junit4:junit4] 2> 121998 T94 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 122001 T106 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33198__collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:33198_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33198",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 122003 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 122017 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 122055 T10 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 48148
[junit4:junit4] 2> 122055 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=10345424
[junit4:junit4] 2> 122189 T34 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 122189 T34 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 123057 T10 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 123058 T10 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 123060 T10 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1ba6db0
[junit4:junit4] 2> 123069 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=40,cumulative_deletesById=20,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 123069 T10 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 123070 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 123070 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 123071 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 123072 T10 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 123072 T10 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2;done=false>>]
[junit4:junit4] 2> 123072 T10 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2
[junit4:junit4] 2> 123073 T10 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index;done=false>>]
[junit4:junit4] 2> 123073 T10 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371252838077/jetty2/index
[junit4:junit4] 2> 123074 T106 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89866426002964490-127.0.0.1:48148_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 123097 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 123156 T10 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 123159 T10 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48085 48085
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=A5AF8CA1B1A42A9C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_CL -Dtests.timezone=US/Eastern -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 123s J0 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<209> but was:<40>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([A5AF8CA1B1A42A9C:244902B9C6FB4AA0]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 123440 T10 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 122669 T9 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 124375 T94 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 124376 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 124376 T94 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {text=PostingsFormat(name=MockRandom), a_t=PostingsFormat(name=Direct), intDefault=PostingsFormat(name=MockSep), other_tl1=PostingsFormat(name=MockSep), range_facet_si=PostingsFormat(name=Direct), a_si=PostingsFormat(name=MockSep), rnd_b=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_sl=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_l=PostingsFormat(name=Direct), _version_=PostingsFormat(name=MockSep), id=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Direct)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=DefaultSimilarity, locale=es_CL, timezone=US/Eastern
[junit4:junit4] 2> NOTE: Linux 3.2.0-45-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=25161336,total=65077248
[junit4:junit4] 2> NOTE: All tests run in this JVM: [RecoveryZkTest]
[junit4:junit4] Completed on J0 in 124.61s, 1 test, 1 failure <<< FAILURES!
[...truncated 877 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:385: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:365: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 38 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure