You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/24 12:58:36 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 6215
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6215/
Java: 64bit/jdk1.6.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
ERROR: SolrIndexSearcher opens=42 closes=41
Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=42 closes=41
at __randomizedtesting.SeedInfo.seed([ADB8E3744056DA70]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:257)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:104)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Build Log:
[...truncated 8973 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 442947 T1630 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 442950 T1630 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1372070778843
[junit4:junit4] 2> 442951 T1630 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 442951 T1631 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 443051 T1630 oasc.ZkTestServer.run start zk server on port:41876
[junit4:junit4] 2> 443053 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443057 T1637 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73f7abc6 name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443058 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443058 T1630 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 443064 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443066 T1639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ddc38b3 name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443067 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443067 T1630 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 443072 T1630 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 443075 T1630 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 443078 T1630 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 443081 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 443082 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 443088 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 443089 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 443093 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 443094 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 443098 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 443098 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 443102 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 443103 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 443107 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 443108 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 443112 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 443112 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 443116 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 443117 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 443134 T1630 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 443135 T1630 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 443226 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 443227 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55254
[junit4:junit4] 2> 443228 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 443228 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 443228 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032
[junit4:junit4] 2> 443229 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/solr.xml
[junit4:junit4] 2> 443229 T1630 oasc.CoreContainer.<init> New CoreContainer 696397115
[junit4:junit4] 2> 443229 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/'
[junit4:junit4] 2> 443230 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/'
[junit4:junit4] 2> 443269 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 443270 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 443270 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 443271 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 443271 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 443271 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 443272 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 443272 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 443272 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 443272 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 443276 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 443276 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 443276 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 443277 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443279 T1650 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a462d19 name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443279 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443280 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 443283 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443284 T1652 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6717f3cb name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443285 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443286 T1630 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 443289 T1630 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 443291 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 443293 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55254_
[junit4:junit4] 2> 443294 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55254_
[junit4:junit4] 2> 443296 T1630 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 443301 T1630 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 443304 T1630 oasc.Overseer.start Overseer (id=89920030562385923-127.0.0.1:55254_-n_0000000000) starting
[junit4:junit4] 2> 443307 T1630 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 443311 T1654 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 443312 T1630 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 443315 T1630 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 443318 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 443322 T1653 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 443326 T1655 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 443326 T1655 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 444826 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 444827 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55254_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55254"}
[junit4:junit4] 2> 444827 T1653 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 444828 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 444831 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 445328 T1655 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/collection1
[junit4:junit4] 2> 445329 T1655 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 445330 T1655 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 445330 T1655 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 445333 T1655 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/collection1/'
[junit4:junit4] 2> 445334 T1655 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/collection1/lib/README' to classloader
[junit4:junit4] 2> 445334 T1655 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 445395 T1655 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 445434 T1655 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 445435 T1655 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 445439 T1655 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 445836 T1655 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 445837 T1655 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 445837 T1655 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 445843 T1655 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 445846 T1655 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 445863 T1655 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 445868 T1655 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 445872 T1655 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 445873 T1655 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 445873 T1655 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 445874 T1655 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 445875 T1655 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 445875 T1655 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 445875 T1655 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 445885 T1655 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372070779032/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/control/data/
[junit4:junit4] 2> 445886 T1655 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 445886 T1655 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 445887 T1655 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/control/data
[junit4:junit4] 2> 445887 T1655 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/control/data/index/
[junit4:junit4] 2> 445888 T1655 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 445888 T1655 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/control/data/index
[junit4:junit4] 2> 445892 T1655 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 445892 T1655 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 445893 T1655 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 445894 T1655 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 445894 T1655 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 445894 T1655 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 445895 T1655 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 445895 T1655 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 445895 T1655 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 445896 T1655 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 445896 T1655 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 445898 T1655 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 445900 T1655 oass.SolrIndexSearcher.<init> Opening Searcher@1dd42575 main
[junit4:junit4] 2> 445901 T1655 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 445901 T1655 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 445905 T1656 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dd42575 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 445905 T1655 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 445906 T1655 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55254 collection:control_collection shard:shard1
[junit4:junit4] 2> 445906 T1655 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 445911 T1655 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 445912 T1655 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 445912 T1655 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 445913 T1655 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55254/collection1/
[junit4:junit4] 2> 445913 T1655 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 445913 T1655 oasc.SyncStrategy.syncToMe http://127.0.0.1:55254/collection1/ has no replicas
[junit4:junit4] 2> 445913 T1655 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55254/collection1/
[junit4:junit4] 2> 445913 T1655 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 446337 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446347 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 446368 T1655 oasc.ZkController.register We are http://127.0.0.1:55254/collection1/ and leader is http://127.0.0.1:55254/collection1/
[junit4:junit4] 2> 446369 T1655 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55254
[junit4:junit4] 2> 446370 T1655 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 446370 T1655 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 446370 T1655 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 446374 T1655 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446375 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 446376 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 446377 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 446384 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 446386 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446388 T1659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31b87d8c name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446388 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446391 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 446396 T1630 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 446483 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 446485 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44979
[junit4:junit4] 2> 446485 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 446486 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 446486 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288
[junit4:junit4] 2> 446486 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/solr.xml
[junit4:junit4] 2> 446486 T1630 oasc.CoreContainer.<init> New CoreContainer 1689163908
[junit4:junit4] 2> 446487 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/'
[junit4:junit4] 2> 446487 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/'
[junit4:junit4] 2> 446550 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 446550 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 446551 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 446551 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 446551 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 446552 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 446552 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 446552 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 446553 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 446553 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 446556 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 446556 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 446557 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 446558 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446560 T1670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78e763da name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446560 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446563 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 446570 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446572 T1672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27080fae name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446572 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446578 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 447582 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44979_
[junit4:junit4] 2> 447584 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44979_
[junit4:junit4] 2> 447587 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 447589 T1659 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 447589 T1672 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 447591 T1652 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 447599 T1673 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 447600 T1673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 447854 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 447856 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:55254__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55254_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55254"}
[junit4:junit4] 2> 447861 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44979_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44979"}
[junit4:junit4] 2> 447861 T1653 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 447861 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 447869 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 447870 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 447869 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 448602 T1673 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/collection1
[junit4:junit4] 2> 448603 T1673 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 448604 T1673 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 448605 T1673 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 448606 T1673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/collection1/'
[junit4:junit4] 2> 448607 T1673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/collection1/lib/README' to classloader
[junit4:junit4] 2> 448608 T1673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 448654 T1673 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 448705 T1673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 448707 T1673 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 448713 T1673 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 449140 T1673 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 449141 T1673 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 449141 T1673 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 449164 T1673 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 449168 T1673 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 449181 T1673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 449185 T1673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 449189 T1673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 449190 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 449190 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 449190 T1673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 449191 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 449191 T1673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 449191 T1673 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 449192 T1673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372070782288/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/
[junit4:junit4] 2> 449192 T1673 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 449192 T1673 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 449193 T1673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1
[junit4:junit4] 2> 449193 T1673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index/
[junit4:junit4] 2> 449193 T1673 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 449194 T1673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index
[junit4:junit4] 2> 449196 T1673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 449197 T1673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 449198 T1673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 449198 T1673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 449199 T1673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 449200 T1673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 449200 T1673 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 449200 T1673 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 449201 T1673 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 449201 T1673 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 449202 T1673 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 449204 T1673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 449208 T1673 oass.SolrIndexSearcher.<init> Opening Searcher@29955a9c main
[junit4:junit4] 2> 449209 T1673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 449209 T1673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 449213 T1674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29955a9c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 449217 T1673 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 449217 T1673 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44979 collection:collection1 shard:shard1
[junit4:junit4] 2> 449218 T1673 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 449223 T1673 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 449225 T1673 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 449225 T1673 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 449225 T1673 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44979/collection1/
[junit4:junit4] 2> 449225 T1673 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 449226 T1673 oasc.SyncStrategy.syncToMe http://127.0.0.1:44979/collection1/ has no replicas
[junit4:junit4] 2> 449226 T1673 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44979/collection1/
[junit4:junit4] 2> 449226 T1673 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 449378 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449391 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 449391 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 449391 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 449430 T1673 oasc.ZkController.register We are http://127.0.0.1:44979/collection1/ and leader is http://127.0.0.1:44979/collection1/
[junit4:junit4] 2> 449430 T1673 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44979
[junit4:junit4] 2> 449430 T1673 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 449431 T1673 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 449431 T1673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 449432 T1673 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449433 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 449434 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 449434 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 449504 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 449505 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51368
[junit4:junit4] 2> 449506 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 449506 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 449507 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329
[junit4:junit4] 2> 449507 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/solr.xml
[junit4:junit4] 2> 449507 T1630 oasc.CoreContainer.<init> New CoreContainer 387066020
[junit4:junit4] 2> 449507 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/'
[junit4:junit4] 2> 449508 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/'
[junit4:junit4] 2> 449559 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 449559 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 449560 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 449560 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 449560 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 449561 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 449561 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 449561 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 449562 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 449562 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 449565 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 449566 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 449566 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 449567 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449568 T1686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@605d5c9c name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 449568 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 449571 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 449576 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449578 T1688 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75ca14ec name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 449578 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 449583 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 450587 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51368_
[junit4:junit4] 2> 450589 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51368_
[junit4:junit4] 2> 450592 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 450594 T1659 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 450594 T1688 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 450594 T1672 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 450594 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 450596 T1652 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 450595 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 450603 T1689 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 450604 T1689 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 450899 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 450900 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44979__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44979_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44979"}
[junit4:junit4] 2> 450906 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51368_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51368"}
[junit4:junit4] 2> 450906 T1653 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 450907 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 450913 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 450913 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 450913 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 450913 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451606 T1689 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/collection1
[junit4:junit4] 2> 451606 T1689 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 451607 T1689 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 451608 T1689 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 451610 T1689 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/collection1/'
[junit4:junit4] 2> 451611 T1689 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/collection1/lib/README' to classloader
[junit4:junit4] 2> 451611 T1689 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 451647 T1689 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 451686 T1689 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 451687 T1689 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 451692 T1689 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 452072 T1689 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 452073 T1689 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 452074 T1689 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 452079 T1689 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 452081 T1689 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 452095 T1689 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 452099 T1689 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 452102 T1689 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 452104 T1689 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 452104 T1689 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 452104 T1689 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 452106 T1689 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 452106 T1689 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 452106 T1689 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 452107 T1689 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372070785329/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/
[junit4:junit4] 2> 452107 T1689 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 452107 T1689 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 452108 T1689 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2
[junit4:junit4] 2> 452108 T1689 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index/
[junit4:junit4] 2> 452108 T1689 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 452108 T1689 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index
[junit4:junit4] 2> 452114 T1689 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 452114 T1689 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 452118 T1689 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 452118 T1689 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 452119 T1689 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 452120 T1689 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 452121 T1689 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 452121 T1689 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 452122 T1689 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 452123 T1689 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 452124 T1689 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 452128 T1689 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 452131 T1689 oass.SolrIndexSearcher.<init> Opening Searcher@2b4d855 main
[junit4:junit4] 2> 452132 T1689 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 452132 T1689 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 452137 T1690 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b4d855 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 452139 T1689 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 452140 T1689 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51368 collection:collection1 shard:shard2
[junit4:junit4] 2> 452152 T1689 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 452161 T1689 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 452163 T1689 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 452163 T1689 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 452164 T1689 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51368/collection1/
[junit4:junit4] 2> 452164 T1689 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 452164 T1689 oasc.SyncStrategy.syncToMe http://127.0.0.1:51368/collection1/ has no replicas
[junit4:junit4] 2> 452164 T1689 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51368/collection1/
[junit4:junit4] 2> 452164 T1689 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 452421 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452432 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452432 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452432 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452432 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452471 T1689 oasc.ZkController.register We are http://127.0.0.1:51368/collection1/ and leader is http://127.0.0.1:51368/collection1/
[junit4:junit4] 2> 452472 T1689 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51368
[junit4:junit4] 2> 452472 T1689 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 452472 T1689 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 452473 T1689 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 452475 T1689 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452477 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 452478 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 452479 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 452565 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 452567 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36754
[junit4:junit4] 2> 452567 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 452568 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 452568 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378
[junit4:junit4] 2> 452568 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/solr.xml
[junit4:junit4] 2> 452568 T1630 oasc.CoreContainer.<init> New CoreContainer 1665267759
[junit4:junit4] 2> 452569 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/'
[junit4:junit4] 2> 452569 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/'
[junit4:junit4] 2> 452610 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 452610 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 452610 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 452611 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 452611 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 452611 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 452612 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 452612 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 452612 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 452613 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 452616 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 452616 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 452617 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 452617 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 452619 T1702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57ada944 name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452619 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 452621 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 452624 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 452625 T1704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72d49876 name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452625 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 452628 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 453630 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36754_
[junit4:junit4] 2> 453633 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36754_
[junit4:junit4] 2> 453636 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 453638 T1659 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453638 T1688 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453638 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453638 T1672 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453639 T1704 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453638 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453639 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453640 T1652 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453647 T1705 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 453647 T1705 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 453940 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 453941 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51368__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51368_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51368"}
[junit4:junit4] 2> 453946 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36754_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36754"}
[junit4:junit4] 2> 453947 T1653 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 453947 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 453953 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453953 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453954 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453953 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453953 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 454649 T1705 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/collection1
[junit4:junit4] 2> 454649 T1705 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 454650 T1705 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 454651 T1705 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 454652 T1705 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/collection1/'
[junit4:junit4] 2> 454654 T1705 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/collection1/lib/README' to classloader
[junit4:junit4] 2> 454654 T1705 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 454720 T1705 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 454755 T1705 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 454756 T1705 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 454762 T1705 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 455170 T1705 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 455171 T1705 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 455172 T1705 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 455178 T1705 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 455180 T1705 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 455193 T1705 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 455197 T1705 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 455200 T1705 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 455201 T1705 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 455202 T1705 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 455202 T1705 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 455203 T1705 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 455204 T1705 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 455204 T1705 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 455205 T1705 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372070788378/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/
[junit4:junit4] 2> 455205 T1705 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 455206 T1705 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 455207 T1705 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3
[junit4:junit4] 2> 455208 T1705 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index/
[junit4:junit4] 2> 455208 T1705 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 455209 T1705 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index
[junit4:junit4] 2> 455213 T1705 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 455214 T1705 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 455216 T1705 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 455217 T1705 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 455218 T1705 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 455219 T1705 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 455219 T1705 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 455220 T1705 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 455220 T1705 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 455221 T1705 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 455222 T1705 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 455224 T1705 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 455229 T1705 oass.SolrIndexSearcher.<init> Opening Searcher@e7d488a main
[junit4:junit4] 2> 455230 T1705 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 455231 T1705 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 455238 T1706 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e7d488a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 455242 T1705 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 455242 T1705 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36754 collection:collection1 shard:shard3
[junit4:junit4] 2> 455243 T1705 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 455251 T1705 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 455254 T1705 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 455255 T1705 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 455255 T1705 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36754/collection1/
[junit4:junit4] 2> 455255 T1705 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 455256 T1705 oasc.SyncStrategy.syncToMe http://127.0.0.1:36754/collection1/ has no replicas
[junit4:junit4] 2> 455256 T1705 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36754/collection1/
[junit4:junit4] 2> 455256 T1705 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 455462 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 455472 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455473 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455473 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455473 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455473 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455513 T1705 oasc.ZkController.register We are http://127.0.0.1:36754/collection1/ and leader is http://127.0.0.1:36754/collection1/
[junit4:junit4] 2> 455514 T1705 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36754
[junit4:junit4] 2> 455514 T1705 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 455515 T1705 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 455515 T1705 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 455517 T1705 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 455519 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 455520 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 455521 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 455603 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 455605 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40743
[junit4:junit4] 2> 455605 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 455606 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 455606 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420
[junit4:junit4] 2> 455606 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/solr.xml
[junit4:junit4] 2> 455606 T1630 oasc.CoreContainer.<init> New CoreContainer 780430955
[junit4:junit4] 2> 455607 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/'
[junit4:junit4] 2> 455607 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/'
[junit4:junit4] 2> 455674 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 455675 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 455675 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 455675 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 455676 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 455676 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 455676 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 455676 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 455677 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 455677 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 455680 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 455681 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 455681 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 455682 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 455683 T1718 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a148faa name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 455684 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 455685 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 455688 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 455689 T1720 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68842c79 name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 455690 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 455693 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 456697 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40743_
[junit4:junit4] 2> 456699 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40743_
[junit4:junit4] 2> 456701 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 456703 T1688 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456704 T1704 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456704 T1720 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456704 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456704 T1672 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456704 T1659 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456705 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456704 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456706 T1652 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456705 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456713 T1721 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 456713 T1721 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 456981 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 456982 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:36754__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36754_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36754"}
[junit4:junit4] 2> 456987 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40743_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40743"}
[junit4:junit4] 2> 456987 T1653 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 456988 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 456995 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456995 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456995 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456995 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456995 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456996 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457715 T1721 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/collection1
[junit4:junit4] 2> 457715 T1721 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 457717 T1721 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 457717 T1721 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 457719 T1721 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/collection1/'
[junit4:junit4] 2> 457719 T1721 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/collection1/lib/README' to classloader
[junit4:junit4] 2> 457720 T1721 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 457761 T1721 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 457806 T1721 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 457807 T1721 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 457812 T1721 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 458211 T1721 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 458212 T1721 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 458213 T1721 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 458218 T1721 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 458221 T1721 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 458237 T1721 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 458241 T1721 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 458245 T1721 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 458247 T1721 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 458247 T1721 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 458247 T1721 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 458249 T1721 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 458249 T1721 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 458250 T1721 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 458250 T1721 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372070791420/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/
[junit4:junit4] 2> 458250 T1721 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 458251 T1721 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 458251 T1721 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4
[junit4:junit4] 2> 458252 T1721 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index/
[junit4:junit4] 2> 458252 T1721 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 458252 T1721 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index
[junit4:junit4] 2> 458256 T1721 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 458257 T1721 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 458261 T1721 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 458261 T1721 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 458262 T1721 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 458263 T1721 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 458264 T1721 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 458265 T1721 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 458265 T1721 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 458266 T1721 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 458267 T1721 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 458270 T1721 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 458273 T1721 oass.SolrIndexSearcher.<init> Opening Searcher@19647f24 main
[junit4:junit4] 2> 458273 T1721 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 458273 T1721 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 458279 T1722 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19647f24 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 458282 T1721 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 458282 T1721 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40743 collection:collection1 shard:shard1
[junit4:junit4] 2> 458287 T1721 oasc.ZkController.register We are http://127.0.0.1:40743/collection1/ and leader is http://127.0.0.1:44979/collection1/
[junit4:junit4] 2> 458287 T1721 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40743
[junit4:junit4] 2> 458288 T1721 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 458288 T1721 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1435 name=collection1 org.apache.solr.core.SolrCore@2cdf406c url=http://127.0.0.1:40743/collection1 node=127.0.0.1:40743_ C1435_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:40743_, base_url=http://127.0.0.1:40743}
[junit4:junit4] 2> 458289 T1723 C1435 P40743 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 458290 T1721 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 458290 T1723 C1435 P40743 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 458291 T1723 C1435 P40743 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 458291 T1723 C1435 P40743 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 458292 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 458293 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 458293 T1723 C1435 P40743 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 458293 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 458302 T1667 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:40743__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 458368 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 458370 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39220
[junit4:junit4] 2> 458371 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 458371 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 458371 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192
[junit4:junit4] 2> 458372 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/solr.xml
[junit4:junit4] 2> 458372 T1630 oasc.CoreContainer.<init> New CoreContainer 1670092317
[junit4:junit4] 2> 458372 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/'
[junit4:junit4] 2> 458373 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/'
[junit4:junit4] 2> 458423 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 458424 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 458424 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 458425 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 458425 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 458425 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 458426 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 458426 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 458426 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 458427 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 458430 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 458430 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 458431 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 458432 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 458433 T1735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24204328 name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 458434 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 458436 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 458441 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 458442 T1737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@458b0684 name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 458443 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 458447 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 458502 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 458503 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40743__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40743_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40743"}
[junit4:junit4] 2> 458507 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458507 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458507 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458507 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458507 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458507 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458507 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 459302 T1667 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:40743__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 459303 T1667 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:40743__collection1&state=recovering&nodeName=127.0.0.1:40743_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 459450 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39220_
[junit4:junit4] 2> 459452 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39220_
[junit4:junit4] 2> 459455 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 459456 T1704 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459456 T1688 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459457 T1659 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459457 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 459457 T1737 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459456 T1672 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459456 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 459456 T1720 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459459 T1652 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 459458 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 459458 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 459457 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 459459 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 459468 T1738 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 459468 T1738 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 460011 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 460012 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39220_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39220"}
[junit4:junit4] 2> 460012 T1653 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 460012 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 460015 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460015 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460015 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460015 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460015 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460015 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460015 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 460470 T1738 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/collection1
[junit4:junit4] 2> 460470 T1738 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 460471 T1738 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 460472 T1738 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 460474 T1738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/collection1/'
[junit4:junit4] 2> 460475 T1738 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/collection1/lib/README' to classloader
[junit4:junit4] 2> 460476 T1738 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 460518 T1738 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 460561 T1738 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 460563 T1738 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 460568 T1738 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 460960 T1738 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 460961 T1738 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 460962 T1738 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 460967 T1738 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 460969 T1738 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 460986 T1738 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 460992 T1738 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 461009 T1738 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 461011 T1738 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 461011 T1738 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 461011 T1738 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 461013 T1738 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 461013 T1738 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 461013 T1738 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 461013 T1738 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372070794192/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/
[junit4:junit4] 2> 461014 T1738 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 461014 T1738 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 461014 T1738 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5
[junit4:junit4] 2> 461015 T1738 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index/
[junit4:junit4] 2> 461015 T1738 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 461015 T1738 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index
[junit4:junit4] 2> 461018 T1738 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 461018 T1738 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 461020 T1738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 461020 T1738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 461021 T1738 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 461022 T1738 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 461022 T1738 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 461022 T1738 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 461023 T1738 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 461023 T1738 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 461024 T1738 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 461026 T1738 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 461030 T1738 oass.SolrIndexSearcher.<init> Opening Searcher@6bbbef48 main
[junit4:junit4] 2> 461031 T1738 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 461031 T1738 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 461037 T1739 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6bbbef48 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 461040 T1738 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 461040 T1738 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39220 collection:collection1 shard:shard2
[junit4:junit4] 2> 461044 T1738 oasc.ZkController.register We are http://127.0.0.1:39220/collection1/ and leader is http://127.0.0.1:51368/collection1/
[junit4:junit4] 2> 461045 T1738 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39220
[junit4:junit4] 2> 461045 T1738 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 461045 T1738 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1436 name=collection1 org.apache.solr.core.SolrCore@11cd40b6 url=http://127.0.0.1:39220/collection1 node=127.0.0.1:39220_ C1436_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:39220_, base_url=http://127.0.0.1:39220}
[junit4:junit4] 2> 461046 T1740 C1436 P39220 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 461046 T1740 C1436 P39220 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 461046 T1738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 461047 T1740 C1436 P39220 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 461047 T1740 C1436 P39220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 461048 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 461048 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 461049 T1740 C1436 P39220 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 461049 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 461058 T1682 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:39220__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 461127 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 461129 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60575
[junit4:junit4] 2> 461129 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 461130 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 461130 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947
[junit4:junit4] 2> 461130 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/solr.xml
[junit4:junit4] 2> 461131 T1630 oasc.CoreContainer.<init> New CoreContainer 2104327529
[junit4:junit4] 2> 461131 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/'
[junit4:junit4] 2> 461131 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/'
[junit4:junit4] 2> 461171 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 461172 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 461172 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 461172 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 461173 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 461173 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 461173 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 461173 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 461174 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 461174 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 461177 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 461178 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 461178 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 461179 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 461180 T1752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58c27655 name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 461180 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 461182 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 461185 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 461186 T1754 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e18e91b name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 461186 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 461189 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1437 name=collection1 org.apache.solr.core.SolrCore@2cdf406c url=http://127.0.0.1:40743/collection1 node=127.0.0.1:40743_ C1437_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:40743_, base_url=http://127.0.0.1:40743}
[junit4:junit4] 2> 461305 T1723 C1437 P40743 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:44979/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 461305 T1723 C1437 P40743 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40743 START replicas=[http://127.0.0.1:44979/collection1/] nUpdates=100
[junit4:junit4] 2> 461306 T1723 C1437 P40743 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 461306 T1723 C1437 P40743 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 461306 T1723 C1437 P40743 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 461307 T1723 C1437 P40743 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 461307 T1723 C1437 P40743 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 461307 T1723 C1437 P40743 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:44979/collection1/. core=collection1
[junit4:junit4] 2> 461307 T1723 C1437 P40743 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1438 name=collection1 org.apache.solr.core.SolrCore@1355725b url=http://127.0.0.1:44979/collection1 node=127.0.0.1:44979_ C1438_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:44979_, base_url=http://127.0.0.1:44979, leader=true}
[junit4:junit4] 2> 461329 T1668 C1438 P44979 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 461334 T1665 C1438 P44979 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 461338 T1665 C1438 P44979 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 461339 T1665 C1438 P44979 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 461352 T1665 C1438 P44979 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 461353 T1665 C1438 P44979 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 461354 T1665 C1438 P44979 oass.SolrIndexSearcher.<init> Opening Searcher@147409e6 realtime
[junit4:junit4] 2> 461355 T1665 C1438 P44979 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 461355 T1665 C1438 P44979 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> ASYNC NEW_CORE C1439 name=collection1 org.apache.solr.core.SolrCore@2cdf406c url=http://127.0.0.1:40743/collection1 node=127.0.0.1:40743_ C1439_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:40743_, base_url=http://127.0.0.1:40743}
[junit4:junit4] 2> 461356 T1723 C1439 P40743 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 461357 T1723 C1439 P40743 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 461359 T1667 C1438 P44979 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 461360 T1667 C1438 P44979 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 461360 T1723 C1439 P40743 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 461361 T1723 C1439 P40743 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 461361 T1723 C1439 P40743 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 461364 T1668 C1438 P44979 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 461364 T1723 C1439 P40743 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 461365 T1723 C1439 P40743 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index.20130624034637257
[junit4:junit4] 2> 461366 T1723 C1439 P40743 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index.20130624034637257 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b84a8be fullCopy=false
[junit4:junit4] 2> 461369 T1665 C1438 P44979 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 461371 T1723 C1439 P40743 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 461383 T1723 C1439 P40743 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 461383 T1723 C1439 P40743 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 461387 T1723 C1439 P40743 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 461388 T1723 C1439 P40743 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 461388 T1723 C1439 P40743 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 461389 T1723 C1439 P40743 oass.SolrIndexSearcher.<init> Opening Searcher@7bd7d288 main
[junit4:junit4] 2> 461391 T1722 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7bd7d288 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 461392 T1723 C1439 P40743 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index.20130624034637257 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index.20130624034637257;done=true>>]
[junit4:junit4] 2> 461392 T1723 C1439 P40743 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index.20130624034637257
[junit4:junit4] 2> 461393 T1723 C1439 P40743 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty4/index.20130624034637257
[junit4:junit4] 2> 461393 T1723 C1439 P40743 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 461394 T1723 C1439 P40743 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 461394 T1723 C1439 P40743 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 461394 T1723 C1439 P40743 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 461397 T1723 C1439 P40743 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 461519 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 461520 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:39220__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39220_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39220"}
[junit4:junit4] 2> 461522 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40743__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40743_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40743"}
[junit4:junit4] 2> 461542 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 461542 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 462059 T1682 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:39220__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 462059 T1682 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:39220__collection1&state=recovering&nodeName=127.0.0.1:39220_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 462192 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60575_
[junit4:junit4] 2> 462194 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60575_
[junit4:junit4] 2> 462196 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 462197 T1754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 462197 T1704 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462197 T1720 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462198 T1737 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462198 T1659 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462199 T1688 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462198 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462199 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462199 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462198 T1672 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462198 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462198 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462201 T1754 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462200 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462200 T1652 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 462209 T1757 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 462209 T1757 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 463050 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 463051 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60575_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60575"}
[junit4:junit4] 2> 463052 T1653 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 463052 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 463058 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463058 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463059 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463058 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463058 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463059 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463059 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463058 T1754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 463211 T1757 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/collection1
[junit4:junit4] 2> 463212 T1757 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 463213 T1757 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 463213 T1757 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 463215 T1757 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/collection1/'
[junit4:junit4] 2> 463216 T1757 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/collection1/lib/README' to classloader
[junit4:junit4] 2> 463217 T1757 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 463264 T1757 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 463305 T1757 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 463307 T1757 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 463312 T1757 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 463690 T1757 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 463690 T1757 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 463691 T1757 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 463696 T1757 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 463699 T1757 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 463709 T1757 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 463712 T1757 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 463714 T1757 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 463715 T1757 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 463715 T1757 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 463715 T1757 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 463716 T1757 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 463716 T1757 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 463717 T1757 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 463717 T1757 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372070796947/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/
[junit4:junit4] 2> 463717 T1757 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6da0d866
[junit4:junit4] 2> 463717 T1757 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 463718 T1757 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6
[junit4:junit4] 2> 463718 T1757 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index/
[junit4:junit4] 2> 463726 T1757 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 463726 T1757 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index
[junit4:junit4] 2> 463729 T1757 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 463730 T1757 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 463731 T1757 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 463732 T1757 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 463732 T1757 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 463732 T1757 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 463733 T1757 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 463733 T1757 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 463733 T1757 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 463734 T1757 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 463734 T1757 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 463736 T1757 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 463738 T1757 oass.SolrIndexSearcher.<init> Opening Searcher@20124588 main
[junit4:junit4] 2> 463739 T1757 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 463739 T1757 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 463743 T1758 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20124588 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 463745 T1757 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 463745 T1757 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60575 collection:collection1 shard:shard3
[junit4:junit4] 2> 463748 T1757 oasc.ZkController.register We are http://127.0.0.1:60575/collection1/ and leader is http://127.0.0.1:36754/collection1/
[junit4:junit4] 2> 463748 T1757 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60575
[junit4:junit4] 2> 463748 T1757 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 463748 T1757 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1440 name=collection1 org.apache.solr.core.SolrCore@51ff24f9 url=http://127.0.0.1:60575/collection1 node=127.0.0.1:60575_ C1440_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:60575_, base_url=http://127.0.0.1:60575}
[junit4:junit4] 2> 463749 T1759 C1440 P60575 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 463749 T1759 C1440 P60575 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 463750 T1757 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 463750 T1759 C1440 P60575 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 463750 T1759 C1440 P60575 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 463750 T1630 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 463751 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 463751 T1759 C1440 P60575 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 463751 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 463754 T1697 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:60575__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 463819 T1630 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 463821 T1630 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38027
[junit4:junit4] 2> 463821 T1630 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 463822 T1630 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 463822 T1630 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372070799645
[junit4:junit4] 2> 463822 T1630 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372070799645/solr.xml
[junit4:junit4] 2> 463823 T1630 oasc.CoreContainer.<init> New CoreContainer 1507237883
[junit4:junit4] 2> 463823 T1630 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372070799645/'
[junit4:junit4] 2> 463823 T1630 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372070799645/'
[junit4:junit4] 2> 463866 T1630 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 463866 T1630 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 463866 T1630 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 463867 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 463867 T1630 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 463867 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 463868 T1630 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 463868 T1630 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 463868 T1630 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 463869 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 463871 T1630 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 463872 T1630 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41876/solr
[junit4:junit4] 2> 463872 T1630 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 463873 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 463874 T1771 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2aeea97c name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 463874 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 463876 T1630 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 463878 T1630 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 463879 T1773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6aaac5dd name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 463879 T1630 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 463883 T1630 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1441 name=collection1 org.apache.solr.core.SolrCore@11cd40b6 url=http://127.0.0.1:39220/collection1 node=127.0.0.1:39220_ C1441_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:39220_, base_url=http://127.0.0.1:39220}
[junit4:junit4] 2> 464061 T1740 C1441 P39220 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:51368/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 464061 T1740 C1441 P39220 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39220 START replicas=[http://127.0.0.1:51368/collection1/] nUpdates=100
[junit4:junit4] 2> 464062 T1740 C1441 P39220 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 464062 T1740 C1441 P39220 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 464062 T1740 C1441 P39220 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 464062 T1740 C1441 P39220 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 464063 T1740 C1441 P39220 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 464063 T1740 C1441 P39220 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:51368/collection1/. core=collection1
[junit4:junit4] 2> 464063 T1740 C1441 P39220 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1442 name=collection1 org.apache.solr.core.SolrCore@59d559f3 url=http://127.0.0.1:51368/collection1 node=127.0.0.1:51368_ C1442_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:51368_, base_url=http://127.0.0.1:51368, leader=true}
[junit4:junit4] 2> 464067 T1681 C1442 P51368 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 464079 T1684 C1442 P51368 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464081 T1684 C1442 P51368 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 464081 T1684 C1442 P51368 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 464092 T1684 C1442 P51368 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 464093 T1684 C1442 P51368 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 464093 T1684 C1442 P51368 oass.SolrIndexSearcher.<init> Opening Searcher@34ffb90a realtime
[junit4:junit4] 2> 464094 T1684 C1442 P51368 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464094 T1684 C1442 P51368 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4] 2> ASYNC NEW_CORE C1443 name=collection1 org.apache.solr.core.SolrCore@11cd40b6 url=http://127.0.0.1:39220/collection1 node=127.0.0.1:39220_ C1443_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:39220_, base_url=http://127.0.0.1:39220}
[junit4:junit4] 2> 464095 T1740 C1443 P39220 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 464095 T1740 C1443 P39220 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 464096 T1683 C1442 P51368 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 464097 T1683 C1442 P51368 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 464097 T1740 C1443 P39220 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 464097 T1740 C1443 P39220 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 464097 T1740 C1443 P39220 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 464099 T1681 C1442 P51368 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 464100 T1740 C1443 P39220 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 464100 T1740 C1443 P39220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index.20130624034639992
[junit4:junit4] 2> 464101 T1740 C1443 P39220 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index.20130624034639992 lockFactory=org.apache.lucene.store.NativeFSLockFactory@32e30de8 fullCopy=false
[junit4:junit4] 2> 464103 T1684 C1442 P51368 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 464104 T1740 C1443 P39220 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 464113 T1740 C1443 P39220 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 464113 T1740 C1443 P39220 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 464114 T1740 C1443 P39220 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 464115 T1740 C1443 P39220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 464115 T1740 C1443 P39220 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 464116 T1740 C1443 P39220 oass.SolrIndexSearcher.<init> Opening Searcher@4ab31a55 main
[junit4:junit4] 2> 464116 T1739 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ab31a55 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 464117 T1740 C1443 P39220 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index.20130624034639992 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index.20130624034639992;done=true>>]
[junit4:junit4] 2> 464117 T1740 C1443 P39220 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index.20130624034639992
[junit4:junit4] 2> 464117 T1740 C1443 P39220 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty5/index.20130624034639992
[junit4:junit4] 2> 464118 T1740 C1443 P39220 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 464118 T1740 C1443 P39220 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 464118 T1740 C1443 P39220 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 464118 T1740 C1443 P39220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 464120 T1740 C1443 P39220 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 464565 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 464567 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:60575__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60575_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60575"}
[junit4:junit4] 2> 464570 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:39220__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39220_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39220"}
[junit4:junit4] 2> 464593 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464593 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464755 T1697 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:60575__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 464755 T1697 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:60575__collection1&state=recovering&nodeName=127.0.0.1:60575_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 464885 T1630 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38027_
[junit4:junit4] 2> 464886 T1630 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38027_
[junit4:junit4] 2> 464888 T1773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464888 T1754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464889 T1737 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464889 T1688 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464889 T1659 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464888 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 464890 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 464889 T1672 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464889 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 464889 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 464889 T1720 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464889 T1704 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464891 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 464891 T1754 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464891 T1773 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464891 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 464892 T1652 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 464892 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 464896 T1776 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 464897 T1776 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 466110 T1653 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 466112 T1653 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38027_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38027"}
[junit4:junit4] 2> 466112 T1653 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 466113 T1653 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 466119 T1704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466120 T1720 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466120 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466121 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466120 T1754 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466120 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466119 T1773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466121 T1659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 466121 T1737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> ASYNC NEW_CORE C1444 name=collection1 org.apache.solr.core.SolrCore@51ff24f9 url=http://127.0.0.1:60575/collection1 node=127.0.0.1:60575_ C1444_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:60575_, base_url=http://127.0.0.1:60575}
[junit4:junit4] 2> 466756 T1759 C1444 P60575 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:36754/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 466757 T1759 C1444 P60575 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60575 START replicas=[http://127.0.0.1:36754/collection1/] nUpdates=100
[junit4:junit4] 2> 466757 T1759 C1444 P60575 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 466758 T1759 C1444 P60575 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 466758 T1759 C1444 P60575 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 466759 T1759 C1444 P60575 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 466759 T1759 C1444 P60575 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 466759 T1759 C1444 P60575 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:36754/collection1/. core=collection1
[junit4:junit4] 2> 466759 T1759 C1444 P60575 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1445 name=collection1 org.apache.solr.core.SolrCore@3ecd010e url=http://127.0.0.1:36754/collection1 node=127.0.0.1:36754_ C1445_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36754_, base_url=http://127.0.0.1:36754, leader=true}
[junit4:junit4] 2> 466768 T1698 C1445 P36754 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 466772 T1699 C1445 P36754 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 466776 T1699 C1445 P36754 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 466777 T1699 C1445 P36754 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 466782 T1699 C1445 P36754 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 466783 T1699 C1445 P36754 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 466784 T1699 C1445 P36754 oass.SolrIndexSearcher.<init> Opening Searcher@7bde3bb0 realtime
[junit4:junit4] 2> 466785 T1699 C1445 P36754 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 466785 T1699 C1445 P36754 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 466787 T1759 C1444 P60575 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 466787 T1759 C1444 P60575 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 466789 T1697 C1445 P36754 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 466789 T1697 C1445 P36754 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 466790 T1759 C1444 P60575 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 466790 T1759 C1444 P60575 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 466791 T1759 C1444 P60575 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 466793 T1698 C1445 P36754 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 466794 T1759 C1444 P60575 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 466795 T1759 C1444 P60575 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index.20130624034642686
[junit4:junit4] 2> 466796 T1759 C1444 P60575 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index.20130624034642686 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e137 fullCopy=false
[junit4:junit4] 2> 466801 T1699 C1445 P36754 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 466803 T1759 C1444 P60575 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 466813 T1759 C1444 P60575 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 466814 T1759 C1444 P60575 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 466817 T1759 C1444 P60575 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 466818 T1759 C1444 P60575 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 466819 T1759 C1444 P60575 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 466819 T1759 C1444 P60575 oass.SolrIndexSearcher.<init> Opening Searcher@6d3d292c main
[junit4:junit4] 2> 466821 T1758 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d3d292c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 466822 T1759 C1444 P60575 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index.20130624034642686 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index.20130624034642686;done=true>>]
[junit4:junit4] 2> 466822 T1759 C1444 P60575 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index.20130624034642686
[junit4:junit4] 2> 466823 T1759 C1444 P60575 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372070778843/jetty6/index.20130624034642686
[junit4:junit4] 2> 466823 T1759 C1444 P60575 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 466824 T1759 C1444 P60575 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit
[...truncated too long message...]
Context.rejoinLeaderElection(ElectionContext.java:377)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:170)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:266)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:377)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:170)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:266)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:377)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:170)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:266)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:377)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:170)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:287)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:284)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:284)
[junit4:junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:731)
[junit4:junit4] 2> ... 33 more
[junit4:junit4] 2>
[junit4:junit4] 2> 538735 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 538735 T1916 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 538735 T1916 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 538735 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 538735 T1916 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 538736 T1916 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 658527 T1630 oas.SolrTestCaseJ4.endTrackingSearchers ERROR ERROR: SolrIndexSearcher opens=42 closes=41
[junit4:junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=496), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=496)), sim=DefaultSimilarity, locale=en_GB, timezone=America/Santa_Isabel
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Sun Microsystems Inc. 1.6.0_45 (64-bit)/cpus=8,threads=2,free=90344136,total=161218560
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ClusterStateTest, TestSuggestSpellingConverter, CurrencyFieldXmlFileTest, IndexBasedSpellCheckerTest, DocValuesMultiTest, TestSolrJ, TestCoreDiscovery, TestRandomFaceting, TestCSVLoader, TestDistributedGrouping, SolrIndexConfigTest, TestZkChroot, TestRTGBase, SolrCoreCheckLockOnStartupTest, TestStressLucene, TestHashPartitioner, TestBM25SimilarityFactory, TestCSVResponseWriter, RequiredFieldsTest, TestPropInject, IndexSchemaRuntimeFieldTest, TestJmxIntegration, ZkNodePropsTest, TestMergePolicyConfig, TestBinaryResponseWriter, SortByFunctionTest, TimeZoneUtilsTest, TestDefaultSimilarityFactory, PingRequestHandlerTest, DebugComponentTest, SolrTestCaseJ4Test, SuggesterWFSTTest, TestQuerySenderNoQuery, FileUtilsTest, TestSchemaNameResource, TestSolr4Spatial, TestIndexingPerformance, DocumentAnalysisRequestHandlerTest, OutputWriterTest, HighlighterTest, TestSolrXMLSerializer, CollectionsAPIDistributedZkTest, BasicDistributedZk2Test, OpenExchangeRatesOrgProviderTest, TestPostingsSolrHighlighter, TestLazyCores, TestManagedSchemaFieldResource, TestIndexSearcher, TestSystemIdResolver, TestFieldTypeResource, SolrPluginUtilsTest, TestPartialUpdateDeduplication, TestDynamicFieldCollectionResource, TestLMJelinekMercerSimilarityFactory, TestOmitPositions, URLClassifyProcessorTest, DisMaxRequestHandlerTest, TestBinaryField, TestDFRSimilarityFactory, TestDistributedSearch, UniqFieldsUpdateProcessorFactoryTest, TestWordDelimiterFilterFactory, CopyFieldTest, TestMaxScoreQueryParser, TestArbitraryIndexDir, TestRangeQuery, TestFoldingMultitermQuery, CurrencyFieldOpenExchangeTest, DistanceFunctionTest, TestFieldResource, FullSolrCloudDistribCmdsTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=ADB8E3744056DA70 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_GB -Dtests.timezone=America/Santa_Isabel -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=42 closes=41
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([ADB8E3744056DA70]:0)
[junit4:junit4] > at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:257)
[junit4:junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:104)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] Completed on J1 in 215.62s, 1 test, 1 failure <<< FAILURES!
[...truncated 574 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1256 tests, 1 suite-level error, 13 ignored (7 assumptions)
Total time: 38 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 6216
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6216/
Java: 32bit/jdk1.6.0_45 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<696> but was:<50>
Stack Trace:
java.lang.AssertionError: expected:<696> but was:<50>
at __randomizedtesting.SeedInfo.seed([2A94205A4445CE3:834FCC1DD31B3CDF]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Build Log:
[...truncated 9082 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 1115851 T1894 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 1115855 T1894 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1372076411343
[junit4:junit4] 2> 1115855 T1894 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 1115856 T1895 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 1115956 T1894 oasc.ZkTestServer.run start zk server on port:37392
[junit4:junit4] 2> 1115957 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1115961 T1901 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e07f5f name:ZooKeeperConnection Watcher:127.0.0.1:37392 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1115962 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1115962 T1894 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 1115966 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1115968 T1903 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@743c7f name:ZooKeeperConnection Watcher:127.0.0.1:37392/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1115968 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1115968 T1894 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 1115971 T1894 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 1115973 T1894 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 1115975 T1894 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 1115976 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1115977 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1115979 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 1115980 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 1115982 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1115982 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1115984 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 1115985 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 1115986 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 1115987 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 1115988 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1115989 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1115990 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1115991 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1115993 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1115993 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1115995 T1894 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1115995 T1894 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1116061 T1894 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1116063 T1894 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57954
[junit4:junit4] 2> 1116064 T1894 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1116064 T1894 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1116065 T1894 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485
[junit4:junit4] 2> 1116065 T1894 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/solr.xml
[junit4:junit4] 2> 1116065 T1894 oasc.CoreContainer.<init> New CoreContainer 2094528
[junit4:junit4] 2> 1116066 T1894 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/'
[junit4:junit4] 2> 1116066 T1894 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/'
[junit4:junit4] 2> 1116118 T1894 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1116119 T1894 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1116119 T1894 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1116119 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1116120 T1894 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1116120 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1116121 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1116121 T1894 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1116121 T1894 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1116122 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1116125 T1894 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1116125 T1894 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37392/solr
[junit4:junit4] 2> 1116126 T1894 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1116126 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1116127 T1914 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59084 name:ZooKeeperConnection Watcher:127.0.0.1:37392 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1116128 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1116129 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1116132 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1116133 T1916 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fd87d name:ZooKeeperConnection Watcher:127.0.0.1:37392/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1116133 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1116135 T1894 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 1116137 T1894 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 1116139 T1894 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 1116140 T1894 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57954_
[junit4:junit4] 2> 1116141 T1894 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57954_
[junit4:junit4] 2> 1116143 T1894 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 1116146 T1894 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 1116148 T1894 oasc.Overseer.start Overseer (id=89920399693905923-127.0.0.1:57954_-n_0000000000) starting
[junit4:junit4] 2> 1116150 T1894 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 1116152 T1918 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 1116152 T1894 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 1116154 T1894 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 1116155 T1894 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1116156 T1917 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 1116159 T1919 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1116159 T1919 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1117659 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1117660 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57954_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57954"}
[junit4:junit4] 2> 1117660 T1917 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1117660 T1917 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1117663 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1118161 T1919 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/collection1
[junit4:junit4] 2> 1118161 T1919 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 1118162 T1919 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1118162 T1919 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 1118164 T1919 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/collection1/'
[junit4:junit4] 2> 1118164 T1919 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/collection1/lib/README' to classloader
[junit4:junit4] 2> 1118164 T1919 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1118196 T1919 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1118243 T1919 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1118244 T1919 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1118249 T1919 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1118668 T1919 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 1118668 T1919 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 1118669 T1919 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 1118676 T1919 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1118679 T1919 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1118689 T1919 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1118692 T1919 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1118695 T1919 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1118696 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1118696 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 1118697 T1919 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1118697 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1118698 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 1118698 T1919 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1118698 T1919 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372076411485/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/control/data/
[junit4:junit4] 2> 1118698 T1919 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc0e01
[junit4:junit4] 2> 1118699 T1919 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1118699 T1919 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/control/data
[junit4:junit4] 2> 1118699 T1919 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/control/data/index/
[junit4:junit4] 2> 1118700 T1919 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1372076411343/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1118700 T1919 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/control/data/index
[junit4:junit4] 2> 1118701 T1919 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@182e1ea lockFactory=org.apache.lucene.store.NativeFSLockFactory@1aecd96)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1118701 T1919 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1118703 T1919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1118703 T1919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1118703 T1919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1118704 T1919 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1118704 T1919 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1118705 T1919 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1118705 T1919 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1118705 T1919 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1118705 T1919 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1118707 T1919 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1118710 T1919 oass.SolrIndexSearcher.<init> Opening Searcher@2595c8 main
[junit4:junit4] 2> 1118710 T1919 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/control/data/tlog
[junit4:junit4] 2> 1118711 T1919 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1118711 T1919 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1118715 T1920 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2595c8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1118716 T1919 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1118716 T1919 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57954 collection:control_collection shard:shard1
[junit4:junit4] 2> 1118717 T1919 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 1118721 T1919 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1118723 T1919 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1118723 T1919 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1118723 T1919 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57954/collection1/
[junit4:junit4] 2> 1118723 T1919 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1118723 T1919 oasc.SyncStrategy.syncToMe http://127.0.0.1:57954/collection1/ has no replicas
[junit4:junit4] 2> 1118724 T1919 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57954/collection1/
[junit4:junit4] 2> 1118724 T1919 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 1119166 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1119172 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1119181 T1919 oasc.ZkController.register We are http://127.0.0.1:57954/collection1/ and leader is http://127.0.0.1:57954/collection1/
[junit4:junit4] 2> 1119181 T1919 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57954
[junit4:junit4] 2> 1119182 T1919 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1119182 T1919 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1119182 T1919 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1119183 T1919 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1119184 T1894 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 1119185 T1894 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1119185 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1119188 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 1119189 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1119190 T1923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1950c3f name:ZooKeeperConnection Watcher:127.0.0.1:37392/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1119191 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1119192 T1894 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1119193 T1894 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 1119261 T1894 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1119263 T1894 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52490
[junit4:junit4] 2> 1119263 T1894 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1119263 T1894 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1119264 T1894 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682
[junit4:junit4] 2> 1119264 T1894 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/solr.xml
[junit4:junit4] 2> 1119265 T1894 oasc.CoreContainer.<init> New CoreContainer 12272225
[junit4:junit4] 2> 1119265 T1894 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/'
[junit4:junit4] 2> 1119265 T1894 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/'
[junit4:junit4] 2> 1119324 T1894 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1119324 T1894 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1119324 T1894 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1119325 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1119325 T1894 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1119325 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1119326 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1119326 T1894 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1119326 T1894 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1119327 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1119330 T1894 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1119330 T1894 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37392/solr
[junit4:junit4] 2> 1119331 T1894 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1119331 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1119333 T1934 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3028ea name:ZooKeeperConnection Watcher:127.0.0.1:37392 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1119333 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1119335 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1119338 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1119339 T1936 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ccebea name:ZooKeeperConnection Watcher:127.0.0.1:37392/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1119339 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1119342 T1894 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1120345 T1894 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52490_
[junit4:junit4] 2> 1120347 T1894 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52490_
[junit4:junit4] 2> 1120349 T1916 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1120349 T1923 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1120349 T1936 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1120349 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1120354 T1937 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1120354 T1937 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1120677 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1120678 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57954_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57954"}
[junit4:junit4] 2> 1120680 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52490_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52490"}
[junit4:junit4] 2> 1120681 T1917 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 1120681 T1917 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1120685 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1120685 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1120685 T1936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1121355 T1937 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/collection1
[junit4:junit4] 2> 1121356 T1937 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1121356 T1937 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1121357 T1937 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1121358 T1937 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/collection1/'
[junit4:junit4] 2> 1121358 T1937 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/collection1/lib/README' to classloader
[junit4:junit4] 2> 1121359 T1937 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1121392 T1937 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1121435 T1937 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1121436 T1937 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1121440 T1937 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1121858 T1937 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 1121858 T1937 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 1121859 T1937 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 1121865 T1937 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1121868 T1937 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1121878 T1937 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1121881 T1937 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1121884 T1937 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1121885 T1937 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1121885 T1937 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 1121885 T1937 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1121886 T1937 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1121886 T1937 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 1121886 T1937 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1121887 T1937 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372076414682/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty1/
[junit4:junit4] 2> 1121887 T1937 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc0e01
[junit4:junit4] 2> 1121887 T1937 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1121888 T1937 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty1
[junit4:junit4] 2> 1121888 T1937 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty1/index/
[junit4:junit4] 2> 1121888 T1937 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1121888 T1937 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty1/index
[junit4:junit4] 2> 1121889 T1937 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1722634 lockFactory=org.apache.lucene.store.NativeFSLockFactory@14080eb)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1121890 T1937 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1121891 T1937 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1121892 T1937 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1121892 T1937 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1121893 T1937 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1121893 T1937 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1121893 T1937 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1121894 T1937 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1121894 T1937 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1121894 T1937 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1121896 T1937 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1121899 T1937 oass.SolrIndexSearcher.<init> Opening Searcher@3fa9c0 main
[junit4:junit4] 2> 1121899 T1937 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty1/tlog
[junit4:junit4] 2> 1121900 T1937 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1121900 T1937 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1121904 T1938 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fa9c0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1121906 T1937 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1121906 T1937 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52490 collection:collection1 shard:shard1
[junit4:junit4] 2> 1121907 T1937 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 1121913 T1937 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1121915 T1937 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1121915 T1937 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1121916 T1937 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52490/collection1/
[junit4:junit4] 2> 1121916 T1937 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1121916 T1937 oasc.SyncStrategy.syncToMe http://127.0.0.1:52490/collection1/ has no replicas
[junit4:junit4] 2> 1121916 T1937 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52490/collection1/
[junit4:junit4] 2> 1121916 T1937 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1122188 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1122193 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1122193 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1122193 T1936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1122221 T1937 oasc.ZkController.register We are http://127.0.0.1:52490/collection1/ and leader is http://127.0.0.1:52490/collection1/
[junit4:junit4] 2> 1122221 T1937 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52490
[junit4:junit4] 2> 1122222 T1937 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1122222 T1937 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1122222 T1937 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1122224 T1937 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1122225 T1894 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 1122225 T1894 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1122226 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1122294 T1894 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1122296 T1894 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41340
[junit4:junit4] 2> 1122297 T1894 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1122297 T1894 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1122298 T1894 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716
[junit4:junit4] 2> 1122298 T1894 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/solr.xml
[junit4:junit4] 2> 1122299 T1894 oasc.CoreContainer.<init> New CoreContainer 25571404
[junit4:junit4] 2> 1122299 T1894 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/'
[junit4:junit4] 2> 1122299 T1894 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/'
[junit4:junit4] 2> 1122351 T1894 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1122352 T1894 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1122352 T1894 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1122353 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1122353 T1894 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1122353 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1122354 T1894 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1122354 T1894 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1122354 T1894 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1122355 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1122358 T1894 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1122358 T1894 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37392/solr
[junit4:junit4] 2> 1122358 T1894 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1122359 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1122361 T1950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@93b1a8 name:ZooKeeperConnection Watcher:127.0.0.1:37392 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1122361 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1122362 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1122365 T1894 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1122366 T1952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@be9185 name:ZooKeeperConnection Watcher:127.0.0.1:37392/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1122366 T1894 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1122369 T1894 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1123372 T1894 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41340_
[junit4:junit4] 2> 1123373 T1894 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41340_
[junit4:junit4] 2> 1123375 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1123375 T1936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1123376 T1952 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1123376 T1916 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1123376 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1123377 T1923 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1123377 T1936 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1123382 T1953 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1123382 T1953 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1123697 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1123698 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52490_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52490"}
[junit4:junit4] 2> 1123700 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41340_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41340"}
[junit4:junit4] 2> 1123700 T1917 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 1123701 T1917 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1123703 T1952 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1123703 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1123703 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1123703 T1936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1124383 T1953 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/collection1
[junit4:junit4] 2> 1124383 T1953 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1124384 T1953 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1124384 T1953 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1124385 T1953 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/collection1/'
[junit4:junit4] 2> 1124386 T1953 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/collection1/lib/README' to classloader
[junit4:junit4] 2> 1124386 T1953 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1124419 T1953 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1124462 T1953 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1124463 T1953 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1124469 T1953 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1124900 T1953 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 1124901 T1953 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 1124901 T1953 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 1124908 T1953 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1124911 T1953 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1124922 T1953 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1124925 T1953 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1124928 T1953 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1124929 T1953 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1124929 T1953 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 1124930 T1953 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1124931 T1953 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1124931 T1953 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 1124931 T1953 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1124932 T1953 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372076417716/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/
[junit4:junit4] 2> 1124932 T1953 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc0e01
[junit4:junit4] 2> 1124932 T1953 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1124933 T1953 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2
[junit4:junit4] 2> 1124933 T1953 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index/
[junit4:junit4] 2> 1124933 T1953 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1124934 T1953 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index
[junit4:junit4] 2> 1124935 T1953 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1124935 T1953 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1124936 T1953 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1124937 T1953 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1124937 T1953 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1124938 T1953 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1124938 T1953 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1124939 T1953 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1124939 T1953 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1124939 T1953 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1124940 T1953 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1124942 T1953 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1124944 T1953 oass.SolrIndexSearcher.<init> Opening Searcher@f145d0 main
[junit4:junit4] 2> 1124944 T1953 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/tlog
[junit4:junit4] 2> 1124945 T1953 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1124945 T1953 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1124950 T1954 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f145d0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1124952 T1953 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1124952 T1953 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41340 collection:collection1 shard:shard1
[junit4:junit4] 2> 1124955 T1953 oasc.ZkController.register We are http://127.0.0.1:41340/collection1/ and leader is http://127.0.0.1:52490/collection1/
[junit4:junit4] 2> 1124956 T1953 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41340
[junit4:junit4] 2> 1124956 T1953 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 1124956 T1953 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C763 name=collection1 org.apache.solr.core.SolrCore@19b758e url=http://127.0.0.1:41340/collection1 node=127.0.0.1:41340_ C763_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41340_, base_url=http://127.0.0.1:41340}
[junit4:junit4] 2> 1124956 T1955 C763 P41340 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 1124957 T1953 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1124957 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 1124957 T1955 C763 P41340 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 1124957 T1955 C763 P41340 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1124958 T1894 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 1124958 T1894 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1124959 T1894 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1124959 T1955 C763 P41340 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1124962 T1894 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1124963 T1930 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C764 name=collection1 org.apache.solr.core.SolrCore@166b536 url=http://127.0.0.1:57954/collection1 node=127.0.0.1:57954_ C764_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57954_, base_url=http://127.0.0.1:57954, leader=true}
[junit4:junit4] 2> 1124970 T1912 C764 P57954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@182e1ea lockFactory=org.apache.lucene.store.NativeFSLockFactory@1aecd96)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1124971 T1912 C764 P57954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1124972 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1438726404657119232)]} 0 3
[junit4:junit4] 2> 1124973 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438726404660264960)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C765 name=collection1 org.apache.solr.core.SolrCore@1e70a1 url=http://127.0.0.1:52490/collection1 node=127.0.0.1:52490_ C765_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52490_, base_url=http://127.0.0.1:52490, leader=true}
[junit4:junit4] 2> 1124981 T1932 C765 P52490 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1722634 lockFactory=org.apache.lucene.store.NativeFSLockFactory@14080eb)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1124982 T1932 C765 P52490 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1124984 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438726404668653568)]} 0 4
[junit4:junit4] 2> 1124984 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1438726404669702144)]} 0 3
[junit4:junit4] 2> 1125065 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1438726404756733952)]} 0 1
[junit4:junit4] 2> 1125065 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438726404756733953)]} 0 1
[junit4:junit4] 2> 1125068 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438726404759879681)]} 0 1
[junit4:junit4] 2> 1125068 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1438726404759879680)]} 0 1
[junit4:junit4] 2> 1125122 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438726404817551360)]} 0 0
[junit4:junit4] 2> 1125123 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1438726404817551361)]} 0 1
[junit4:junit4] 2> 1125125 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1438726404820697088)]} 0 1
[junit4:junit4] 2> 1125125 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438726404819648512)]} 0 1
[junit4:junit4] 2> 1125207 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1125208 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41340_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41340"}
[junit4:junit4] 2> 1125210 T1936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1125211 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1125211 T1952 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1125210 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1125225 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1438726404925554688)]} 0 0
[junit4:junit4] 2> 1125226 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1438726404925554689)]} 0 1
[junit4:junit4] 2> C763_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41340_, base_url=http://127.0.0.1:41340}
[junit4:junit4] 2> 1125235 T1946 C763 P41340 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1125236 T1946 C763 P41340 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1125237 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10001 (-1438726404928700416)]} 0 3
[junit4:junit4] 2> 1125237 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[1 (-1438726404928700417)]} 0 0
[junit4:junit4] 2> 1125237 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1438726404928700416)]} 0 9
[junit4:junit4] 2> 1125238 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1438726404928700417)]} 0 10
[junit4:junit4] 2> 1125241 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1438726404941283328)]} 0 1
[junit4:junit4] 2> 1125241 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1438726404941283329)]} 0 1
[junit4:junit4] 2> 1125246 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10004 (1438726404944429056)]} 0 1
[junit4:junit4] 2> 1125247 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1438726404944429057)]} 0 2
[junit4:junit4] 2> 1125247 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1438726404944429056)]} 0 4
[junit4:junit4] 2> 1125247 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1438726404944429057)]} 0 4
[junit4:junit4] 2> 1125276 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1438726404977983489)]} 0 1
[junit4:junit4] 2> 1125276 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1438726404977983488)]} 0 1
[junit4:junit4] 2> 1125282 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10005 (1438726404981129216)]} 0 1
[junit4:junit4] 2> 1125282 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1438726404981129216)]} 0 4
[junit4:junit4] 2> 1125282 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1438726404982177792)]} 0 1
[junit4:junit4] 2> 1125283 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1438726404982177792)]} 0 5
[junit4:junit4] 2> 1125307 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1438726405010489344)]} 0 1
[junit4:junit4] 2> 1125308 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1438726405011537920)]} 0 1
[junit4:junit4] 2> 1125313 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10006 (1438726405013635072)]} 0 1
[junit4:junit4] 2> 1125313 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1438726405013635072)]} 0 4
[junit4:junit4] 2> 1125313 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1438726405014683648)]} 0 0
[junit4:junit4] 2> 1125314 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1438726405014683648)]} 0 4
[junit4:junit4] 2> 1125317 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10003 (-1438726405022023680)]} 0 0
[junit4:junit4] 2> 1125318 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[3 (-1438726405023072256)]} 0 0
[junit4:junit4] 2> 1125321 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10003 (-1438726405024120832)]} 0 0
[junit4:junit4] 2> 1125321 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10003 (-1438726405024120832)]} 0 2
[junit4:junit4] 2> 1125321 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[3 (-1438726405025169408)]} 0 0
[junit4:junit4] 2> 1125322 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[3 (-1438726405025169408)]} 0 2
[junit4:junit4] 2> 1125323 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1438726405028315136)]} 0 0
[junit4:junit4] 2> 1125325 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1438726405029363712)]} 0 1
[junit4:junit4] 2> 1125328 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10007 (1438726405030412288)]} 0 1
[junit4:junit4] 2> 1125328 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1438726405030412288)]} 0 3
[junit4:junit4] 2> 1125329 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1438726405032509440)]} 0 1
[junit4:junit4] 2> 1125330 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1438726405032509440)]} 0 3
[junit4:junit4] 2> 1125331 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10004 (-1438726405036703744)]} 0 0
[junit4:junit4] 2> 1125333 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[4 (-1438726405038800896)]} 0 0
[junit4:junit4] 2> 1125334 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10004 (-1438726405038800896)]} 0 0
[junit4:junit4] 2> 1125335 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10004 (-1438726405038800896)]} 0 2
[junit4:junit4] 2> 1125336 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[4 (-1438726405040898048)]} 0 0
[junit4:junit4] 2> 1125337 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[4 (-1438726405040898048)]} 0 2
[junit4:junit4] 2> 1125337 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1438726405042995200)]} 0 0
[junit4:junit4] 2> 1125339 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1438726405045092352)]} 0 0
[junit4:junit4] 2> 1125341 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10008 (1438726405045092352)]} 0 0
[junit4:junit4] 2> 1125342 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1438726405045092352)]} 0 3
[junit4:junit4] 2> 1125344 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1438726405047189504)]} 0 1
[junit4:junit4] 2> 1125344 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1438726405047189504)]} 0 3
[junit4:junit4] 2> 1125363 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1438726405069209600)]} 0 1
[junit4:junit4] 2> 1125365 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1438726405071306752)]} 0 1
[junit4:junit4] 2> 1125368 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10009 (1438726405073403904)]} 0 0
[junit4:junit4] 2> 1125369 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1438726405073403904)]} 0 3
[junit4:junit4] 2> 1125371 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1438726405074452480)]} 0 1
[junit4:junit4] 2> 1125371 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1438726405074452480)]} 0 4
[junit4:junit4] 2> 1125417 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1438726405126881280)]} 0 0
[junit4:junit4] 2> 1125420 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1438726405130027008)]} 0 0
[junit4:junit4] 2> 1125421 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10006 (-1438726405128978432)]} 0 0
[junit4:junit4] 2> 1125421 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1438726405128978432)]} 0 2
[junit4:junit4] 2> 1125423 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[6 (-1438726405132124160)]} 0 0
[junit4:junit4] 2> 1125424 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1438726405132124160)]} 0 2
[junit4:junit4] 2> 1125424 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1438726405133172736)]} 0 1
[junit4:junit4] 2> 1125426 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1438726405136318464)]} 0 0
[junit4:junit4] 2> 1125428 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10010 (1438726405136318464)]} 0 0
[junit4:junit4] 2> 1125429 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1438726405136318464)]} 0 3
[junit4:junit4] 2> 1125431 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1438726405138415616)]} 0 1
[junit4:junit4] 2> 1125431 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1438726405138415616)]} 0 3
[junit4:junit4] 2> 1125474 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1438726405186650112)]} 0 0
[junit4:junit4] 2> 1125476 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1438726405188747264)]} 0 0
[junit4:junit4] 2> 1125478 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10007 (-1438726405188747264)]} 0 0
[junit4:junit4] 2> 1125478 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1438726405188747264)]} 0 2
[junit4:junit4] 2> 1125479 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[7 (-1438726405190844416)]} 0 0
[junit4:junit4] 2> 1125480 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1438726405190844416)]} 0 2
[junit4:junit4] 2> 1125481 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1438726405192941568)]} 0 1
[junit4:junit4] 2> 1125482 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1438726405193990144)]} 0 1
[junit4:junit4] 2> 1125485 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10011 (1438726405196087296)]} 0 0
[junit4:junit4] 2> 1125486 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1438726405196087296)]} 0 3
[junit4:junit4] 2> 1125486 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1438726405197135872)]} 0 0
[junit4:junit4] 2> 1125487 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1438726405197135872)]} 0 3
[junit4:junit4] 2> 1125501 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1438726405214961664)]} 0 0
[junit4:junit4] 2> 1125502 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1438726405216010240)]} 0 1
[junit4:junit4] 2> 1125506 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1438726405218107392)]} 0 0
[junit4:junit4] 2> 1125507 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10012 (1438726405217058816)]} 0 1
[junit4:junit4] 2> 1125507 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1438726405218107392)]} 0 3
[junit4:junit4] 2> 1125507 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1438726405217058816)]} 0 4
[junit4:junit4] 2> 1125600 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[8 (-1438726405318770689)]} 0 0
[junit4:junit4] 2> 1125600 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10008 (-1438726405318770688)]} 0 0
[junit4:junit4] 2> 1125604 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[8 (-1438726405320867840)]} 0 0
[junit4:junit4] 2> 1125604 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10008 (-1438726405321916416)]} 0 0
[junit4:junit4] 2> 1125604 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[8 (-1438726405320867840)]} 0 2
[junit4:junit4] 2> 1125605 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10008 (-1438726405321916416)]} 0 2
[junit4:junit4] 2> 1125607 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1438726405325062144)]} 0 1
[junit4:junit4] 2> 1125607 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1438726405326110720)]} 0 0
[junit4:junit4] 2> 1125611 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1438726405328207872)]} 0 0
[junit4:junit4] 2> 1125612 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10013 (1438726405328207873)]} 0 1
[junit4:junit4] 2> 1125612 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1438726405328207872)]} 0 3
[junit4:junit4] 2> 1125612 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1438726405328207873)]} 0 3
[junit4:junit4] 2> 1125638 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1438726405357568000)]} 0 1
[junit4:junit4] 2> 1125639 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1438726405358616576)]} 0 1
[junit4:junit4] 2> 1125643 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1438726405360713728)]} 0 1
[junit4:junit4] 2> 1125643 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1438726405360713729)]} 0 1
[junit4:junit4] 2> 1125643 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1438726405360713728)]} 0 3
[junit4:junit4] 2> 1125643 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1438726405360713729)]} 0 3
[junit4:junit4] 2> 1125727 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1438726405451939840)]} 0 0
[junit4:junit4] 2> 1125728 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1438726405451939841)]} 0 1
[junit4:junit4] 2> 1125732 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1438726405454036992)]} 0 1
[junit4:junit4] 2> 1125732 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1438726405454036993)]} 0 1
[junit4:junit4] 2> 1125732 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1438726405454036992)]} 0 3
[junit4:junit4] 2> 1125732 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1438726405454036993)]} 0 3
[junit4:junit4] 2> 1125780 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1438726405506465792)]} 0 1
[junit4:junit4] 2> 1125780 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1438726405507514368)]} 0 0
[junit4:junit4] 2> 1125785 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1438726405509611520)]} 0 1
[junit4:junit4] 2> 1125785 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1438726405509611521)]} 0 0
[junit4:junit4] 2> 1125785 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1438726405509611520)]} 0 3
[junit4:junit4] 2> 1125786 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1438726405509611521)]} 0 4
[junit4:junit4] 2> 1125821 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10 (-1438726405550505984)]} 0 0
[junit4:junit4] 2> 1125821 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10010 (-1438726405550505985)]} 0 0
[junit4:junit4] 2> 1125824 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10 (-1438726405552603136)]} 0 0
[junit4:junit4] 2> 1125824 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10010 (-1438726405552603137)]} 0 0
[junit4:junit4] 2> 1125824 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10 (-1438726405552603136)]} 0 2
[junit4:junit4] 2> 1125825 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10010 (-1438726405552603137)]} 0 2
[junit4:junit4] 2> 1125827 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1438726405555748864)]} 0 1
[junit4:junit4] 2> 1125828 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1438726405556797440)]} 0 1
[junit4:junit4] 2> 1125831 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1438726405558894592)]} 0 0
[junit4:junit4] 2> 1125832 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1438726405558894593)]} 0 1
[junit4:junit4] 2> 1125832 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1438726405558894592)]} 0 3
[junit4:junit4] 2> 1125832 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1438726405558894593)]} 0 3
[junit4:junit4] 2> 1125889 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1438726405620760576)]} 0 1
[junit4:junit4] 2> 1125889 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1438726405620760577)]} 0 1
[junit4:junit4] 2> 1125893 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1438726405623906304)]} 0 0
[junit4:junit4] 2> 1125894 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1438726405623906305)]} 0 1
[junit4:junit4] 2> 1125894 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1438726405623906304)]} 0 3
[junit4:junit4] 2> 1125894 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1438726405623906305)]} 0 3
[junit4:junit4] 2> 1125953 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10011 (-1438726405688918017)]} 0 0
[junit4:junit4] 2> 1125953 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[11 (-1438726405688918016)]} 0 0
[junit4:junit4] 2> 1125957 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10011 (-1438726405691015168)]} 0 0
[junit4:junit4] 2> 1125957 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[11 (-1438726405691015169)]} 0 0
[junit4:junit4] 2> 1125957 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10011 (-1438726405691015168)]} 0 2
[junit4:junit4] 2> 1125958 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[11 (-1438726405691015169)]} 0 3
[junit4:junit4] 2> 1125960 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1438726405695209472)]} 0 1
[junit4:junit4] 2> 1125960 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1438726405696258048)]} 0 0
[junit4:junit4] 2> 1125963 T1930 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 1125963 T1930 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:41340_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4] 2> 1125965 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1438726405698355200)]} 0 2
[junit4:junit4] 2> 1125965 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1438726405698355201)]} 0 1
[junit4:junit4] 2> 1125965 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1438726405698355200)]} 0 3
[junit4:junit4] 2> 1125966 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1438726405698355201)]} 0 4
[junit4:junit4] 2> 1126045 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[13 (-1438726405785387008)]} 0 0
[junit4:junit4] 2> 1126045 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10013 (-1438726405785387009)]} 0 0
[junit4:junit4] 2> 1126048 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[13 (-1438726405787484160)]} 0 0
[junit4:junit4] 2> 1126048 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10013 (-1438726405787484161)]} 0 0
[junit4:junit4] 2> 1126049 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[13 (-1438726405787484160)]} 0 2
[junit4:junit4] 2> 1126049 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10013 (-1438726405787484161)]} 0 2
[junit4:junit4] 2> 1126051 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1438726405791678464)]} 0 0
[junit4:junit4] 2> 1126052 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1438726405791678465)]} 0 1
[junit4:junit4] 2> 1126056 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1438726405793775616)]} 0 1
[junit4:junit4] 2> 1126056 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1438726405793775616)]} 0 3
[junit4:junit4] 2> 1126056 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1438726405794824192)]} 0 0
[junit4:junit4] 2> 1126057 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1438726405794824192)]} 0 3
[junit4:junit4] 2> 1126148 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1438726405892341760)]} 0 1
[junit4:junit4] 2> 1126148 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1438726405893390336)]} 0 0
[junit4:junit4] 2> 1126153 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1438726405895487488)]} 0 1
[junit4:junit4] 2> 1126153 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1438726405895487489)]} 0 0
[junit4:junit4] 2> 1126153 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1438726405895487488)]} 0 3
[junit4:junit4] 2> 1126154 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1438726405895487489)]} 0 4
[junit4:junit4] 2> 1126220 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[16 (-1438726405968887808)]} 0 0
[junit4:junit4] 2> 1126220 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10016 (-1438726405968887809)]} 0 0
[junit4:junit4] 2> 1126223 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[16 (-1438726405970984960)]} 0 0
[junit4:junit4] 2> 1126224 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10016 (-1438726405970984961)]} 0 0
[junit4:junit4] 2> 1126224 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[16 (-1438726405970984960)]} 0 2
[junit4:junit4] 2> 1126224 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10016 (-1438726405970984961)]} 0 2
[junit4:junit4] 2> 1126226 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1438726405975179264)]} 0 0
[junit4:junit4] 2> 1126227 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10022 (1438726405975179265)]} 0 1
[junit4:junit4] 2> 1126231 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1438726405977276416)]} 0 1
[junit4:junit4] 2> 1126231 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1438726405978324992)]} 0 1
[junit4:junit4] 2> 1126231 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1438726405977276416)]} 0 3
[junit4:junit4] 2> 1126232 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10022 (1438726405978324992)]} 0 3
[junit4:junit4] 2> 1126261 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1438726406011879424)]} 0 0
[junit4:junit4] 2> 1126261 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10023 (1438726406011879425)]} 0 0
[junit4:junit4] 2> 1126265 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1438726406013976576)]} 0 0
[junit4:junit4] 2> 1126266 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1438726406013976576)]} 0 3
[junit4:junit4] 2> 1126266 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1438726406013976577)]} 0 1
[junit4:junit4] 2> 1126267 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10023 (1438726406013976577)]} 0 4
[junit4:junit4] 2> 1126336 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[19 (-1438726406090522624)]} 0 0
[junit4:junit4] 2> 1126337 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10019 (-1438726406090522625)]} 0 1
[junit4:junit4] 2> 1126340 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[19 (-1438726406092619776)]} 0 0
[junit4:junit4] 2> 1126340 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10019 (-1438726406092619777)]} 0 0
[junit4:junit4] 2> 1126340 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[19 (-1438726406092619776)]} 0 2
[junit4:junit4] 2> 1126341 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10019 (-1438726406092619777)]} 0 3
[junit4:junit4] 2> 1126343 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1438726406096814080)]} 0 1
[junit4:junit4] 2> 1126343 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10024 (1438726406097862656)]} 0 0
[junit4:junit4] 2> 1126347 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1438726406099959808)]} 0 0
[junit4:junit4] 2> 1126348 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1438726406099959809)]} 0 1
[junit4:junit4] 2> 1126348 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1438726406099959808)]} 0 3
[junit4:junit4] 2> 1126348 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10024 (1438726406099959809)]} 0 3
[junit4:junit4] 2> 1126394 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1438726406150291456)]} 0 1
[junit4:junit4] 2> 1126394 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10025 (1438726406151340032)]} 0 1
[junit4:junit4] 2> 1126399 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1438726406153437184)]} 0 1
[junit4:junit4] 2> 1126399 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1438726406153437185)]} 0 1
[junit4:junit4] 2> 1126399 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1438726406153437184)]} 0 3
[junit4:junit4] 2> 1126400 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10025 (1438726406153437185)]} 0 4
[junit4:junit4] 2> 1126424 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[21 (-1438726406182797312)]} 0 0
[junit4:junit4] 2> 1126425 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10021 (-1438726406183845888)]} 0 1
[junit4:junit4] 2> 1126428 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[21 (-1438726406185943040)]} 0 0
[junit4:junit4] 2> 1126428 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10021 (-1438726406185943041)]} 0 0
[junit4:junit4] 2> 1126429 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[21 (-1438726406185943040)]} 0 3
[junit4:junit4] 2> 1126429 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10021 (-1438726406185943041)]} 0 2
[junit4:junit4] 2> 1126431 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1438726406190137344)]} 0 0
[junit4:junit4] 2> 1126432 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10026 (1438726406190137345)]} 0 1
[junit4:junit4] 2> 1126436 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1438726406192234496)]} 0 1
[junit4:junit4] 2> 1126436 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1438726406193283072)]} 0 1
[junit4:junit4] 2> 1126436 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1438726406192234496)]} 0 3
[junit4:junit4] 2> 1126437 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10026 (1438726406193283072)]} 0 3
[junit4:junit4] 2> 1126446 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1438726406204817408)]} 0 1
[junit4:junit4] 2> 1126446 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10027 (1438726406205865984)]} 0 0
[junit4:junit4] 2> 1126450 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1438726406207963136)]} 0 0
[junit4:junit4] 2> 1126451 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1438726406207963136)]} 0 3
[junit4:junit4] 2> 1126451 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1438726406207963137)]} 0 1
[junit4:junit4] 2> 1126451 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10027 (1438726406207963137)]} 0 3
[junit4:junit4] 2> 1126454 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1438726406214254592)]} 0 0
[junit4:junit4] 2> 1126455 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10028 (1438726406214254593)]} 0 1
[junit4:junit4] 2> 1126458 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1438726406216351744)]} 0 0
[junit4:junit4] 2> 1126459 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1438726406216351744)]} 0 3
[junit4:junit4] 2> 1126459 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1438726406217400320)]} 0 1
[junit4:junit4] 2> 1126459 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10028 (1438726406217400320)]} 0 2
[junit4:junit4] 2> 1126554 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1438726406318063616)]} 0 1
[junit4:junit4] 2> 1126554 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10029 (1438726406318063617)]} 0 1
[junit4:junit4] 2> 1126558 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1438726406321209344)]} 0 0
[junit4:junit4] 2> 1126559 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1438726406321209345)]} 0 1
[junit4:junit4] 2> 1126559 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1438726406321209344)]} 0 4
[junit4:junit4] 2> 1126559 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10029 (1438726406321209345)]} 0 3
[junit4:junit4] 2> 1126604 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10023 (-1438726406370492416)]} 0 1
[junit4:junit4] 2> 1126604 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[23 (-1438726406370492417)]} 0 1
[junit4:junit4] 2> 1126607 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10023 (-1438726406373638144)]} 0 0
[junit4:junit4] 2> 1126607 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[23 (-1438726406373638145)]} 0 0
[junit4:junit4] 2> 1126608 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10023 (-1438726406373638144)]} 0 2
[junit4:junit4] 2> 1126608 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[23 (-1438726406373638145)]} 0 2
[junit4:junit4] 2> 1126611 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10030 (1438726406377832448)]} 0 1
[junit4:junit4] 2> 1126611 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1438726406377832449)]} 0 1
[junit4:junit4] 2> 1126616 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1438726406380978176)]} 0 1
[junit4:junit4] 2> 1126617 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1438726406380978177)]} 0 1
[junit4:junit4] 2> 1126617 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10030 (1438726406380978176)]} 0 4
[junit4:junit4] 2> 1126617 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1438726406380978177)]} 0 4
[junit4:junit4] 2> 1126691 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10031 (1438726406461718528)]} 0 1
[junit4:junit4] 2> 1126691 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1438726406461718529)]} 0 1
[junit4:junit4] 2> 1126695 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1438726406464864256)]} 0 0
[junit4:junit4] 2> 1126696 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10031 (1438726406464864256)]} 0 3
[junit4:junit4] 2> 1126696 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1438726406464864257)]} 0 1
[junit4:junit4] 2> 1126697 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1438726406464864257)]} 0 4
[junit4:junit4] 2> 1126742 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10032 (1438726406515195904)]} 0 1
[junit4:junit4] 2> 1126742 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1438726406516244480)]} 0 0
[junit4:junit4] 2> 1126747 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1438726406518341632)]} 0 1
[junit4:junit4] 2> 1126747 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1438726406518341633)]} 0 1
[junit4:junit4] 2> 1126747 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10032 (1438726406518341632)]} 0 3
[junit4:junit4] 2> 1126748 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1438726406518341633)]} 0 4
[junit4:junit4] 2> 1126764 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10033 (1438726406539313152)]} 0 0
[junit4:junit4] 2> 1126765 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1438726406539313153)]} 0 1
[junit4:junit4] 2> 1126769 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1438726406541410304)]} 0 0
[junit4:junit4] 2> 1126770 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1438726406542458880)]} 0 1
[junit4:junit4] 2> 1126770 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10033 (1438726406541410304)]} 0 4
[junit4:junit4] 2> 1126771 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1438726406542458880)]} 0 4
[junit4:junit4] 2> 1126870 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1438726406650462209)]} 0 0
[junit4:junit4] 2> 1126871 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10034 (1438726406650462208)]} 0 1
[junit4:junit4] 2> 1126875 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1438726406653607936)]} 0 0
[junit4:junit4] 2> 1126876 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1438726406653607937)]} 0 1
[junit4:junit4] 2> 1126876 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1438726406653607936)]} 0 4
[junit4:junit4] 2> 1126877 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10034 (1438726406653607937)]} 0 4
[junit4:junit4] 2> 1126917 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1438726406699745281)]} 0 0
[junit4:junit4] 2> 1126918 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10035 (1438726406699745280)]} 0 1
[junit4:junit4] 2> 1126922 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1438726406701842432)]} 0 1
[junit4:junit4] 2> 1126922 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1438726406701842433)]} 0 1
[junit4:junit4] 2> 1126922 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1438726406701842432)]} 0 3
[junit4:junit4] 2> 1126923 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10035 (1438726406701842433)]} 0 4
[junit4:junit4] 2> 1126964 T1894 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 41340
[junit4:junit4] 2> 1126964 T1894 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=25571404
[junit4:junit4] 2> 1127005 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10025 (-1438726406792019968)]} 0 0
[junit4:junit4] 2> 1127005 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[25 (-1438726406792019969)]} 0 0
[junit4:junit4] 2> 1127009 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10025 (-1438726406794117120)]} 0 0
[junit4:junit4] 2> 1127009 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[25 (-1438726406795165696)]} 0 0
[junit4:junit4] 2> 1127010 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10025 (-1438726406794117120)]} 0 3
[junit4:junit4] 2> 1127010 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[25 (-1438726406795165696)]} 0 3
[junit4:junit4] 2> 1127012 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10036 (1438726406799360000)]} 0 1
[junit4:junit4] 2> 1127013 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1438726406799360001)]} 0 1
[junit4:junit4] 2> 1127017 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1438726406801457152)]} 0 1
[junit4:junit4] 2> 1127017 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1438726406801457153)]} 0 1
[junit4:junit4] 2> 1127017 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10036 (1438726406801457152)]} 0 3
[junit4:junit4] 2> 1127018 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1438726406801457153)]} 0 4
[junit4:junit4] 2> 1127089 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1438726406880100352)]} 0 0
[junit4:junit4] 2> 1127090 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10037 (1438726406880100353)]} 0 1
[junit4:junit4] 2> 1127094 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1438726406883246080)]} 0 1
[junit4:junit4] 2> 1127094 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1438726406883246081)]} 0 0
[junit4:junit4] 2> 1127095 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1438726406883246080)]} 0 4
[junit4:junit4] 2> 1127095 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10037 (1438726406883246081)]} 0 3
[junit4:junit4] 2> 1127184 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[26 (-1438726406979715072)]} 0 0
[junit4:junit4] 2> 1127184 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10026 (-1438726406979715073)]} 0 0
[junit4:junit4] 2> 1127188 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1438726406981812224)]} 0 0
[junit4:junit4] 2> 1127188 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1438726406982860800)]} 0 0
[junit4:junit4] 2> 1127188 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[26 (-1438726406981812224)]} 0 2
[junit4:junit4] 2> 1127189 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10026 (-1438726406982860800)]} 0 3
[junit4:junit4] 2> 1127191 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1438726406986006528)]} 0 1
[junit4:junit4] 2> 1127192 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10038 (1438726406987055104)]} 0 2
[junit4:junit4] 2> 1127195 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1438726406989152256)]} 0 0
[junit4:junit4] 2> 1127196 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1438726406989152256)]} 0 3
[junit4:junit4] 2> 1127196 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1438726406990200832)]} 0 0
[junit4:junit4] 2> 1127197 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10038 (1438726406990200832)]} 0 4
[junit4:junit4] 2> 1127283 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[28 (-1438726407083524096)]} 0 0
[junit4:junit4] 2> 1127284 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10028 (-1438726407084572672)]} 0 0
[junit4:junit4] 2> 1127287 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[28 (-1438726407085621248)]} 0 0
[junit4:junit4] 2> 1127287 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10028 (-1438726407086669824)]} 0 0
[junit4:junit4] 2> 1127288 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[28 (-1438726407085621248)]} 0 3
[junit4:junit4] 2> 1127288 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10028 (-1438726407086669824)]} 0 2
[junit4:junit4] 2> 1127290 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1438726407090864128)]} 0 1
[junit4:junit4] 2> 1127291 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10039 (1438726407090864129)]} 0 1
[junit4:junit4] 2> 1127294 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1438726407092961280)]} 0 0
[junit4:junit4] 2> 1127295 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1438726407092961280)]} 0 3
[junit4:junit4] 2> 1127295 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10039 (1438726407094009856)]} 0 0
[junit4:junit4] 2> 1127296 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10039 (1438726407094009856)]} 0 3
[junit4:junit4] 2> 1127331 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1438726407133855744)]} 0 0
[junit4:junit4] 2> 1127332 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10040 (1438726407133855745)]} 0 1
[junit4:junit4] 2> 1127336 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1438726407135952896)]} 0 0
[junit4:junit4] 2> 1127337 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10040 (1438726407137001472)]} 0 1
[junit4:junit4] 2> 1127337 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1438726407135952896)]} 0 4
[junit4:junit4] 2> 1127337 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10040 (1438726407137001472)]} 0 3
[junit4:junit4] 2> 1127349 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1438726407151681536)]} 0 1
[junit4:junit4] 2> 1127349 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10041 (1438726407151681537)]} 0 1
[junit4:junit4] 2> 1127353 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1438726407154827264)]} 0 0
[junit4:junit4] 2> 1127354 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10041 (1438726407154827265)]} 0 1
[junit4:junit4] 2> 1127354 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1438726407154827264)]} 0 3
[junit4:junit4] 2> 1127354 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10041 (1438726407154827265)]} 0 3
[junit4:junit4] 2> 1127444 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[34 (-1438726407252344833)]} 0 0
[junit4:junit4] 2> 1127444 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10034 (-1438726407252344832)]} 0 0
[junit4:junit4] 2> 1127448 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[34 (-1438726407254441984)]} 0 0
[junit4:junit4] 2> 1127449 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10034 (-1438726407255490560)]} 0 1
[junit4:junit4] 2> 1127449 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[34 (-1438726407254441984)]} 0 3
[junit4:junit4] 2> 1127449 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10034 (-1438726407255490560)]} 0 2
[junit4:junit4] 2> 1127451 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1438726407259684864)]} 0 0
[junit4:junit4] 2> 1127452 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10042 (1438726407259684865)]} 0 1
[junit4:junit4] 2> 1127456 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1438726407261782016)]} 0 1
[junit4:junit4] 2> 1127457 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1438726407261782016)]} 0 4
[junit4:junit4] 2> 1127457 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10042 (1438726407262830592)]} 0 1
[junit4:junit4] 2> 1127457 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10042 (1438726407262830592)]} 0 3
[junit4:junit4] 2> 1127475 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[35 (-1438726407284850688)]} 0 0
[junit4:junit4] 2> 1127475 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10035 (-1438726407284850689)]} 0 0
[junit4:junit4] 2> 1127478 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[35 (-1438726407286947840)]} 0 0
[junit4:junit4] 2> 1127479 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[35 (-1438726407286947840)]} 0 2
[junit4:junit4] 2> 1127479 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10035 (-1438726407286947841)]} 0 0
[junit4:junit4] 2> 1127480 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10035 (-1438726407286947841)]} 0 3
[junit4:junit4] 2> 1127481 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1438726407291142144)]} 0 0
[junit4:junit4] 2> 1127483 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10043 (1438726407292190720)]} 0 1
[junit4:junit4] 2> 1127486 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1438726407294287872)]} 0 1
[junit4:junit4] 2> 1127487 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1438726407294287872)]} 0 4
[junit4:junit4] 2> 1127487 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10043 (1438726407295336448)]} 0 0
[junit4:junit4] 2> 1127488 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10043 (1438726407295336448)]} 0 3
[junit4:junit4] 2> 1127579 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1438726407392854016)]} 0 1
[junit4:junit4] 2> 1127579 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10044 (1438726407393902592)]} 0 0
[junit4:junit4] 2> 1127584 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1438726407395999744)]} 0 1
[junit4:junit4] 2> 1127584 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10044 (1438726407395999745)]} 0 1
[junit4:junit4] 2> 1127584 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1438726407395999744)]} 0 3
[junit4:junit4] 2> 1127584 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10044 (1438726407395999745)]} 0 3
[junit4:junit4] 2> 1127609 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1438726407425359872)]} 0 1
[junit4:junit4] 2> 1127610 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10045 (1438726407425359873)]} 0 1
[junit4:junit4] 2> 1127614 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1438726407428505600)]} 0 0
[junit4:junit4] 2> 1127615 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1438726407428505600)]} 0 3
[junit4:junit4] 2> 1127615 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10045 (1438726407428505601)]} 0 1
[junit4:junit4] 2> 1127615 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10045 (1438726407428505601)]} 0 3
[junit4:junit4] 2> 1127674 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[36 (-1438726407493517312)]} 0 0
[junit4:junit4] 2> 1127674 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10036 (-1438726407493517313)]} 0 0
[junit4:junit4] 2> 1127677 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[36 (-1438726407495614464)]} 0 0
[junit4:junit4] 2> 1127678 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10036 (-1438726407495614465)]} 0 0
[junit4:junit4] 2> 1127678 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[36 (-1438726407495614464)]} 0 2
[junit4:junit4] 2> 1127678 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10036 (-1438726407495614465)]} 0 2
[junit4:junit4] 2> 1127681 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1438726407499808768)]} 0 1
[junit4:junit4] 2> 1127681 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10046 (1438726407500857344)]} 0 1
[junit4:junit4] 2> 1127685 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1438726407502954496)]} 0 0
[junit4:junit4] 2> 1127686 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1438726407502954496)]} 0 3
[junit4:junit4] 2> 1127686 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10046 (1438726407502954497)]} 0 1
[junit4:junit4] 2> 1127687 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10046 (1438726407502954497)]} 0 4
[junit4:junit4] 2> 1127721 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1438726407541751808)]} 0 1
[junit4:junit4] 2> 1127721 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10047 (1438726407542800384)]} 0 0
[junit4:junit4] 2> 1127726 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1438726407544897536)]} 0 1
[junit4:junit4] 2> 1127726 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10047 (1438726407544897537)]} 0 1
[junit4:junit4] 2> 1127727 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1438726407544897536)]} 0 4
[junit4:junit4] 2> 1127727 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10047 (1438726407544897537)]} 0 4
[junit4:junit4] 2> 1127818 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1438726407643463680)]} 0 1
[junit4:junit4] 2> 1127818 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10048 (1438726407643463681)]} 0 1
[junit4:junit4] 2> 1127823 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1438726407646609408)]} 0 1
[junit4:junit4] 2> 1127823 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1438726407646609408)]} 0 3
[junit4:junit4] 2> 1127823 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10048 (1438726407646609409)]} 0 1
[junit4:junit4] 2> 1127824 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10048 (1438726407646609409)]} 0 4
[junit4:junit4] 2> 1127846 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1438726407672823808)]} 0 0
[junit4:junit4] 2> 1127846 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10049 (1438726407672823809)]} 0 1
[junit4:junit4] 2> 1127850 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1438726407675969536)]} 0 0
[junit4:junit4] 2> 1127851 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1438726407675969536)]} 0 3
[junit4:junit4] 2> 1127851 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10049 (1438726407675969537)]} 0 0
[junit4:junit4] 2> 1127852 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10049 (1438726407675969537)]} 0 4
[junit4:junit4] 2> 1127892 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[37 (-1438726407722106880)]} 0 0
[junit4:junit4] 2> 1127893 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10037 (-1438726407723155456)]} 0 0
[junit4:junit4] 2> 1127896 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[37 (-1438726407724204032)]} 0 0
[junit4:junit4] 2> 1127896 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[37 (-1438726407724204032)]} 0 2
[junit4:junit4] 2> 1127896 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10037 (-1438726407725252608)]} 0 0
[junit4:junit4] 2> 1127897 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10037 (-1438726407725252608)]} 0 2
[junit4:junit4] 2> 1127898 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1438726407728398336)]} 0 0
[junit4:junit4] 2> 1127899 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10050 (1438726407729446912)]} 0 0
[junit4:junit4] 2> 1127903 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1438726407730495488)]} 0 1
[junit4:junit4] 2> 1127903 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10050 (1438726407731544064)]} 0 0
[junit4:junit4] 2> 1127904 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1438726407730495488)]} 0 3
[junit4:junit4] 2> 1127904 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10050 (1438726407731544064)]} 0 3
[junit4:junit4] 2> 1127932 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1438726407763001344)]} 0 1
[junit4:junit4] 2> 1127932 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10051 (1438726407764049920)]} 0 0
[junit4:junit4] 2> 1127936 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1438726407766147072)]} 0 0
[junit4:junit4] 2> 1127937 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10051 (1438726407766147073)]} 0 1
[junit4:junit4] 2> 1127937 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1438726407766147072)]} 0 3
[junit4:junit4] 2> 1127937 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10051 (1438726407766147073)]} 0 3
[junit4:junit4] 2> 1127957 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1438726407789215744)]} 0 2
[junit4:junit4] 2> 1127957 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10052 (1438726407789215745)]} 0 1
[junit4:junit4] 2> 1127961 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1438726407792361472)]} 0 0
[junit4:junit4] 2> 1127962 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1438726407792361472)]} 0 3
[junit4:junit4] 2> 1127962 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10052 (1438726407792361473)]} 0 1
[junit4:junit4] 2> 1127963 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10052 (1438726407792361473)]} 0 4
[junit4:junit4] 2> 1127964 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52490/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 1127964 T1955 C763 P41340 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41340 START replicas=[http://127.0.0.1:52490/collection1/] nUpdates=100
[junit4:junit4] 2> 1127965 T1955 C763 P41340 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 1127966 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 1127966 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 1127966 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 1127966 T1929 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 1127966 T1955 C763 P41340 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 1127967 T1955 C763 P41340 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52490/collection1/. core=collection1
[junit4:junit4] 2> 1127967 T1955 C763 P41340 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1127970 T1930 C765 P52490 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1127976 T1930 C765 P52490 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1722634 lockFactory=org.apache.lucene.store.NativeFSLockFactory@14080eb)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1722634 lockFactory=org.apache.lucene.store.NativeFSLockFactory@14080eb)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1127976 T1930 C765 P52490 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1127978 T1930 C765 P52490 oass.SolrIndexSearcher.<init> Opening Searcher@88bcaa realtime
[junit4:junit4] 2> 1127978 T1930 C765 P52490 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1127978 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 1127979 T1955 C763 P41340 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1127979 T1955 C763 P41340 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 1127980 T1932 C765 P52490 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1127981 T1932 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 1127981 T1955 C763 P41340 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 1127981 T1955 C763 P41340 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 1127981 T1955 C763 P41340 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 1127983 T1929 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 1127983 T1955 C763 P41340 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 9
[junit4:junit4] 2> 1127984 T1955 C763 P41340 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index.20130624092023472
[junit4:junit4] 2> 1127984 T1955 C763 P41340 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@eb09e7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c669b6) fullCopy=false
[junit4:junit4] 2> 1127986 T1931 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1127989 T1932 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1127991 T1930 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_1.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1127994 T1929 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1127996 T1931 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1127998 T1932 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1128001 T1930 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_1_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1128003 T1929 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_1.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1128006 T1931 C765 P52490 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_1.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 1128006 T1955 C763 P41340 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 1128007 T1955 C763 P41340 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 1128007 T1955 C763 P41340 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 1128007 T1955 C763 P41340 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 1128011 T1955 C763 P41340 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1128011 T1955 C763 P41340 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1128013 T1955 C763 P41340 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1128013 T1955 C763 P41340 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1128013 T1955 C763 P41340 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 1128014 T1955 C763 P41340 oass.SolrIndexSearcher.<init> Opening Searcher@13a28ae main
[junit4:junit4] 2> 1128014 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[38 (-1438726407850033152)]} 0 0
[junit4:junit4] 2> 1128015 T1954 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13a28ae main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c98/36)}
[junit4:junit4] 2> 1128015 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10038 (-1438726407851081728)]} 0 0
[junit4:junit4] 2> 1128015 T1955 C763 P41340 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index.20130624092023472 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index.20130624092023472;done=true>>]
[junit4:junit4] 2> 1128016 T1955 C763 P41340 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index.20130624092023472
[junit4:junit4] 2> 1128016 T1955 C763 P41340 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index.20130624092023472
[junit4:junit4] 2> 1128016 T1955 C763 P41340 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 1128016 T1963 C763 P41340 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=14024
[junit4:junit4] 2> 1128017 T1963 C763 P41340 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1128017 T1963 C763 P41340 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@207eea lockFactory=org.apache.lucene.store.NativeFSLockFactory@15787fa)),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1128018 T1963 C763 P41340 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1128018 T1963 C763 P41340 oass.SolrIndexSearcher.<init> Opening Searcher@1ea4d42 main
[junit4:junit4] 2> 1128018 T1963 C763 P41340 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1128019 T1954 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ea4d42 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c98/36)}
[junit4:junit4] 2> 1128019 T1963 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] {} 0 3
[junit4:junit4] 2> 1128020 T1963 C763 P41340 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=14024}
[junit4:junit4] 2> 1128020 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 1128020 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[38 (-1438726407852130304)]} 0 2
[junit4:junit4] 2> 1128020 T1955 C763 P41340 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1128021 T1955 C763 P41340 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1128020 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10038 (-1438726407853178880)]} 0 2
[junit4:junit4] 2> 1128021 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[38 (-1438726407852130304)]} 0 5
[junit4:junit4] 2> 1128021 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10038 (-1438726407853178880)]} 0 4
[junit4:junit4] 2> 1128022 T1955 C763 P41340 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 1128024 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1438726407859470336)]} 0 1
[junit4:junit4] 2> 1128025 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10053 (1438726407860518912)]} 0 2
[junit4:junit4] 2> 1128030 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1438726407862616064)]} 0 1
[junit4:junit4] 2> 1128030 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1438726407862616064)]} 0 4
[junit4:junit4] 2> 1128030 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10053 (1438726407863664640)]} 0 1
[junit4:junit4] 2> 1128031 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10053 (1438726407863664640)]} 0 4
[junit4:junit4] 2> 1128074 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1438726407911899136)]} 0 1
[junit4:junit4] 2> 1128075 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10054 (1438726407912947712)]} 0 1
[junit4:junit4] 2> 1128079 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1438726407915044864)]} 0 1
[junit4:junit4] 2> 1128079 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1438726407915044864)]} 0 3
[junit4:junit4] 2> 1128080 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10054 (1438726407915044865)]} 0 1
[junit4:junit4] 2> 1128080 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10054 (1438726407915044865)]} 0 4
[junit4:junit4] 2> 1128084 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[40 (-1438726407923433472)]} 0 0
[junit4:junit4] 2> 1128085 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10040 (-1438726407924482048)]} 0 0
[junit4:junit4] 2> 1128088 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[40 (-1438726407925530624)]} 0 0
[junit4:junit4] 2> 1128088 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[40 (-1438726407925530624)]} 0 2
[junit4:junit4] 2> 1128089 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10040 (-1438726407926579200)]} 0 0
[junit4:junit4] 2> 1128089 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10040 (-1438726407926579200)]} 0 2
[junit4:junit4] 2> 1128091 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1438726407929724928)]} 0 1
[junit4:junit4] 2> 1128092 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10055 (1438726407930773504)]} 0 1
[junit4:junit4] 2> 1128095 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1438726407932870656)]} 0 1
[junit4:junit4] 2> 1128095 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1438726407932870656)]} 0 2
[junit4:junit4] 2> 1128096 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10055 (1438726407933919232)]} 0 0
[junit4:junit4] 2> 1128097 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10055 (1438726407933919232)]} 0 3
[junit4:junit4] 2> 1128150 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1438726407991590912)]} 0 1
[junit4:junit4] 2> 1128151 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10056 (1438726407993688064)]} 0 0
[junit4:junit4] 2> 1128155 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1438726407994736640)]} 0 0
[junit4:junit4] 2> 1128156 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10056 (1438726407995785216)]} 0 1
[junit4:junit4] 2> 1128156 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1438726407994736640)]} 0 4
[junit4:junit4] 2> 1128156 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10056 (1438726407995785216)]} 0 3
[junit4:junit4] 2> 1128160 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1438726408002076672)]} 0 1
[junit4:junit4] 2> 1128161 T1912 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10057 (1438726408003125248)]} 0 1
[junit4:junit4] 2> 1128165 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1438726408005222400)]} 0 1
[junit4:junit4] 2> 1128165 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10057 (1438726408005222401)]} 0 1
[junit4:junit4] 2> 1128165 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1438726408005222400)]} 0 3
[junit4:junit4] 2> 1128166 T1929 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10057 (1438726408005222401)]} 0 4
[junit4:junit4] 2> 1128215 T1917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1128216 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41340_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41340"}
[junit4:junit4] 2> 1128218 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1438726408062894080)]} 0 1
[junit4:junit4] 2> 1128218 T1917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41340_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41340"}
[junit4:junit4] 2> 1128218 T1910 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10058 (1438726408063942656)]} 0 0
[junit4:junit4] 2> 1128221 T1936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1128221 T1952 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1128221 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1128221 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> C763_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41340_, base_url=http://127.0.0.1:41340}
[junit4:junit4] 2> 1128223 T1947 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1438726408066039808)]} 0 1
[junit4:junit4] 2> 1128223 T1948 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10058 (1438726408066039809)]} 0 1
[junit4:junit4] 2> 1128224 T1931 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1438726408066039808)]} 0 4
[junit4:junit4] 2> 1128224 T1930 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10058 (1438726408066039809)]} 0 4
[junit4:junit4] 2> 1128264 T1911 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[42 (-1438726408112177152)]} 0 0
[junit4:junit4] 2> 1128264 T1909 C764 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10042 (-1438726408112177153)]} 0 0
[junit4:junit4] 2> 1128268 T1945 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[42 (-1438726408114274304)]} 0 0
[junit4:junit4] 2> 1128268 T1946 C763 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10042 (-1438726408114274305)]} 0 0
[junit4:junit4] 2> 1128268 T1932 C765 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[42 (-1438726408114274304)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C766 name=collection1 org.apache.solr.core.SolrCore@1e70a1 url=http://127.0.0.1:52490/collection1 node=127.0.0.1:52490_ C766_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52490_, base_url=http://127.0.0.1:52490, leader=true}
[junit4:junit4] 2> 1128269 T1929 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10042 (-1438726408114274305)]} 0 3
[junit4:junit4] 2> ASYNC NEW_CORE C767 name=collection1 org.apache.solr.core.SolrCore@166b536 url=http://127.0.0.1:57954/collection1 node=127.0.0.1:57954_ C767_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57954_, base_url=http://127.0.0.1:57954, leader=true}
[junit4:junit4] 2> 1128274 T1912 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1438726408122662912)]} 0 0
[junit4:junit4] 2> 1128275 T1910 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10059 (1438726408122662913)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C768 name=collection1 org.apache.solr.core.SolrCore@19b758e url=http://127.0.0.1:41340/collection1 node=127.0.0.1:41340_ C768_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41340_, base_url=http://127.0.0.1:41340}
[junit4:junit4] 2> 1128278 T1947 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1438726408124760064)]} 0 0
[junit4:junit4] 2> 1128279 T1931 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1438726408124760064)]} 0 3
[junit4:junit4] 2> 1128279 T1948 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10059 (1438726408124760065)]} 0 1
[junit4:junit4] 2> 1128280 T1930 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10059 (1438726408124760065)]} 0 4
[junit4:junit4] 2> 1128300 T1911 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1438726408148877312)]} 0 1
[junit4:junit4] 2> 1128300 T1909 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10060 (1438726408149925888)]} 0 0
[junit4:junit4] 2> 1128304 T1945 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1438726408152023040)]} 0 1
[junit4:junit4] 2> 1128304 T1932 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1438726408152023040)]} 0 2
[junit4:junit4] 2> 1128305 T1946 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10060 (1438726408152023041)]} 0 1
[junit4:junit4] 2> 1128305 T1929 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10060 (1438726408152023041)]} 0 3
[junit4:junit4] 2> 1128331 T1912 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1438726408182431744)]} 0 0
[junit4:junit4] 2> 1128332 T1910 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10061 (1438726408182431745)]} 0 1
[junit4:junit4] 2> 1128337 T1947 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1438726408185577472)]} 0 1
[junit4:junit4] 2> 1128337 T1948 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10061 (1438726408185577473)]} 0 1
[junit4:junit4] 2> 1128337 T1931 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1438726408185577472)]} 0 3
[junit4:junit4] 2> 1128338 T1930 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10061 (1438726408185577473)]} 0 4
[junit4:junit4] 2> 1128340 T1911 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[44 (-1438726408190820352)]} 0 1
[junit4:junit4] 2> 1128340 T1909 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10044 (-1438726408191868928)]} 0 0
[junit4:junit4] 2> 1128343 T1945 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[44 (-1438726408192917504)]} 0 0
[junit4:junit4] 2> 1128344 T1946 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10044 (-1438726408193966080)]} 0 0
[junit4:junit4] 2> 1128344 T1932 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[44 (-1438726408192917504)]} 0 3
[junit4:junit4] 2> 1128344 T1929 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10044 (-1438726408193966080)]} 0 2
[junit4:junit4] 2> 1128347 T1912 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1438726408198160384)]} 0 1
[junit4:junit4] 2> 1128348 T1910 C767 P57954 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10062 (1438726408199208960)]} 0 1
[junit4:junit4] 2> 1128351 T1947 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1438726408201306112)]} 0 1
[junit4:junit4] 2> 1128352 T1931 C766 P52490 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1438726408201306112)]} 0 4
[junit4:junit4] 2> 1128353 T1947 C768 P41340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:52490/collection1/&update.distrib=FROMLEADER&wt=javabin&version=
[...truncated too long message...]
nit4:junit4] 2> 1250685 T1991 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1250686 T1991 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:52490_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52490"}
[junit4:junit4] 2> 1250688 T1991 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52490_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52490"}
[junit4:junit4] 2> 1250694 T1977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1250724 T1894 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 41340
[junit4:junit4] 2> 1250725 T1894 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22637971
[junit4:junit4] 2> 1250871 T1916 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1250871 T1916 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1250871 T1916 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1251727 T1894 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1251727 T1894 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1251729 T1894 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1b23851
[junit4:junit4] 2> 1251736 T1894 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=56,cumulative_deletesById=20,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 1251736 T1894 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1251737 T1894 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1251737 T1894 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1251738 T1894 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1251738 T1894 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1251739 T1894 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index;done=false>>]
[junit4:junit4] 2> 1251739 T1894 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2/index
[junit4:junit4] 2> 1251739 T1894 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2;done=false>>]
[junit4:junit4] 2> 1251740 T1894 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1372076411343/jetty2
[junit4:junit4] 2> 1251740 T1991 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89920399693905930-127.0.0.1:41340_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 1251741 T1977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1251741 T1977 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1251742 T1977 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1251762 T1894 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 1251816 T1894 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 1251818 T1894 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:37392 37392
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=2A94205A4445CE3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_CR -Dtests.timezone=America/Paramaribo -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 136s J1 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<696> but was:<50>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([2A94205A4445CE3:834FCC1DD31B3CDF]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 1251826 T1894 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 135978 T1893 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=79), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=79)), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_CR, timezone=America/Paramaribo
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=3,free=165303912,total=214564864
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestStressLucene, SortByFunctionTest, TestCoreDiscovery, HardAutoCommitTest, TestDefaultSimilarityFactory, IndexBasedSpellCheckerTest, URLClassifyProcessorTest, DateMathParserTest, TestSolrJ, UpdateParamsTest, TestRecovery, TestQuerySenderNoQuery, SampleTest, PathHierarchyTokenizerFactoryTest, TestOmitPositions, TestDistributedGrouping, TestUtils, SimplePostToolTest, ChaosMonkeySafeLeaderTest, CoreContainerCoreInitFailuresTest, TestRangeQuery, ConvertedLegacyTest, DistanceFunctionTest, TestRandomFaceting, TestStressReorder, TestFieldResource, JSONWriterTest, SuggesterWFSTTest, TestFastOutputStream, DocValuesMultiTest, CollectionsAPIDistributedZkTest, BasicZkTest, TermVectorComponentDistributedTest, DOMUtilTest, TestQueryTypes, TestSolrXml, SpellCheckComponentTest, RequestHandlersTest, TestDefaultSearchFieldResource, PingRequestHandlerTest, TestManagedSchemaFieldResource, TestLuceneMatchVersion, TestDFRSimilarityFactory, TestComponentsName, MultiTermTest, NoCacheHeaderTest, TestAddFieldRealTimeGet, ZkNodePropsTest, DirectUpdateHandlerTest, TestFiltering, TestLMJelinekMercerSimilarityFactory, UniqFieldsUpdateProcessorFactoryTest, TestFieldTypeResource, TestElisionMultitermQuery, TestMergePolicyConfig, IndexSchemaRuntimeFieldTest, TestRTGBase, TestSuggestSpellingConverter, TestBinaryResponseWriter, TestCopyFieldCollectionResource, DirectSolrSpellCheckerTest, StandardRequestHandlerTest, SignatureUpdateProcessorFactoryTest, FileUtilsTest, TestShardHandlerFactory, TestSolr4Spatial, NumericFieldsTest, DocumentAnalysisRequestHandlerTest, SolrIndexConfigTest, HighlighterConfigTest, ShardRoutingTest, RecoveryZkTest]
[junit4:junit4] Completed on J1 in 136.76s, 1 test, 1 failure <<< FAILURES!
[...truncated 414 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1256 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 43 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure