You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/19 04:56:00 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6202 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6202/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:56867/collection1lastClient and got 258 from http://127.0.0.1:47774/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:56867/collection1lastClient and got 258 from http://127.0.0.1:47774/collection1
at __randomizedtesting.SeedInfo.seed([82C33D59EEE16665:325B34199BE0659]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9585 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 629800 T2119 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 629805 T2119 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1371609851229
[junit4:junit4] 2> 629805 T2119 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 629806 T2120 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 629906 T2119 oasc.ZkTestServer.run start zk server on port:50920
[junit4:junit4] 2> 629907 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 630040 T2126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ae47532 name:ZooKeeperConnection Watcher:127.0.0.1:50920 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 630040 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 630041 T2119 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 630045 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 630047 T2128 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@403d53dd name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 630047 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 630047 T2119 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 630050 T2119 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 630052 T2119 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 630054 T2119 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 630056 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 630057 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 630061 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 630061 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 630064 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 630065 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 630067 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 630068 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 630070 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 630071 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 630073 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 630074 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 630076 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 630077 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 630079 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 630080 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 630082 T2119 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 630083 T2119 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 630157 T2119 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 630161 T2119 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40028
[junit4:junit4] 2> 630162 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 630162 T2119 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 630163 T2119 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511
[junit4:junit4] 2> 630163 T2119 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/solr.xml
[junit4:junit4] 2> 630164 T2119 oasc.CoreContainer.<init> New CoreContainer 140169403
[junit4:junit4] 2> 630164 T2119 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/'
[junit4:junit4] 2> 630165 T2119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/'
[junit4:junit4] 2> 630227 T2119 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 630227 T2119 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 630228 T2119 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 630228 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 630228 T2119 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 630228 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 630229 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 630229 T2119 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 630229 T2119 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 630230 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 630233 T2119 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 630233 T2119 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50920/solr
[junit4:junit4] 2> 630234 T2119 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 630234 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 630236 T2139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4724c7f8 name:ZooKeeperConnection Watcher:127.0.0.1:50920 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 630237 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 630238 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 630243 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 630244 T2141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49f586f name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 630244 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 630246 T2119 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 630248 T2119 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 630251 T2119 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 630252 T2119 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40028_
[junit4:junit4] 2> 630253 T2119 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40028_
[junit4:junit4] 2> 630255 T2119 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 630260 T2119 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 630261 T2119 oasc.Overseer.start Overseer (id=89889823210340355-127.0.0.1:40028_-n_0000000000) starting
[junit4:junit4] 2> 630264 T2119 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 630267 T2143 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 630267 T2119 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 630269 T2119 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 630271 T2119 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 630276 T2144 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 630276 T2142 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 630276 T2144 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 631779 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 631780 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40028_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40028"}
[junit4:junit4] 2> 631780 T2142 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 631781 T2142 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 631784 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 632278 T2144 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/collection1
[junit4:junit4] 2> 632278 T2144 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 632279 T2144 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 632279 T2144 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 632280 T2144 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/collection1/'
[junit4:junit4] 2> 632281 T2144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/collection1/lib/README' to classloader
[junit4:junit4] 2> 632282 T2144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 632314 T2144 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 632347 T2144 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 632349 T2144 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 632352 T2144 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 632725 T2144 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 632730 T2144 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 632733 T2144 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 632743 T2144 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 632747 T2144 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 632750 T2144 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 632751 T2144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 632751 T2144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 632751 T2144 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 632752 T2144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 632752 T2144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 632752 T2144 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 632759 T2144 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371609851511/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/
[junit4:junit4] 2> 632759 T2144 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2625e90a
[junit4:junit4] 2> 632759 T2144 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 632760 T2144 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data
[junit4:junit4] 2> 632760 T2144 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index/
[junit4:junit4] 2> 632760 T2144 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 632761 T2144 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index
[junit4:junit4] 2> 632780 T2144 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 632781 T2144 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 632783 T2144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 632783 T2144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 632783 T2144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 632784 T2144 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 632784 T2144 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 632784 T2144 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 632785 T2144 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 632785 T2144 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 632785 T2144 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 632788 T2144 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 632791 T2144 oass.SolrIndexSearcher.<init> Opening Searcher@5743dfdf main
[junit4:junit4] 2> 632792 T2144 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 632792 T2144 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 632796 T2145 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5743dfdf main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 632797 T2144 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 632797 T2144 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40028 collection:control_collection shard:shard1
[junit4:junit4] 2> 632798 T2144 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 632803 T2144 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 632805 T2144 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 632805 T2144 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 632805 T2144 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40028/collection1/
[junit4:junit4] 2> 632805 T2144 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 632805 T2144 oasc.SyncStrategy.syncToMe http://127.0.0.1:40028/collection1/ has no replicas
[junit4:junit4] 2> 632806 T2144 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40028/collection1/
[junit4:junit4] 2> 632806 T2144 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 633287 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 633294 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 633311 T2144 oasc.ZkController.register We are http://127.0.0.1:40028/collection1/ and leader is http://127.0.0.1:40028/collection1/
[junit4:junit4] 2> 633311 T2144 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40028
[junit4:junit4] 2> 633312 T2144 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 633312 T2144 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 633312 T2144 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 633313 T2144 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 633315 T2119 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 633315 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 633316 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 633320 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 633321 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 633323 T2148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7162c54e name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 633323 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 633325 T2119 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 633328 T2119 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 633421 T2119 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 633425 T2119 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56867
[junit4:junit4] 2> 633426 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 633427 T2119 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 633427 T2119 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752
[junit4:junit4] 2> 633428 T2119 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/solr.xml
[junit4:junit4] 2> 633428 T2119 oasc.CoreContainer.<init> New CoreContainer 1663579958
[junit4:junit4] 2> 633429 T2119 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/'
[junit4:junit4] 2> 633429 T2119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/'
[junit4:junit4] 2> 633483 T2119 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 633483 T2119 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 633484 T2119 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 633484 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 633484 T2119 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 633484 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 633485 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 633485 T2119 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 633485 T2119 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 633486 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 633488 T2119 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 633489 T2119 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50920/solr
[junit4:junit4] 2> 633489 T2119 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 633490 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 633492 T2159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b137d73 name:ZooKeeperConnection Watcher:127.0.0.1:50920 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 633492 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 633494 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 633496 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 633515 T2161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cc414b9 name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 633515 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 633518 T2119 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 634521 T2119 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56867_
[junit4:junit4] 2> 634523 T2119 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56867_
[junit4:junit4] 2> 634526 T2161 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 634526 T2141 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 634526 T2148 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 634527 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 634531 T2162 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 634531 T2162 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 634799 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 634800 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40028_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40028"}
[junit4:junit4] 2> 634803 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56867_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56867"}
[junit4:junit4] 2> 634803 T2142 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 634803 T2142 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 634807 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 634807 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 634807 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 635533 T2162 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/collection1
[junit4:junit4] 2> 635533 T2162 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 635534 T2162 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 635534 T2162 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 635535 T2162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/collection1/'
[junit4:junit4] 2> 635536 T2162 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/collection1/lib/README' to classloader
[junit4:junit4] 2> 635536 T2162 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 635570 T2162 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 635613 T2162 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 635615 T2162 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 635618 T2162 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 636010 T2162 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 636024 T2162 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 636027 T2162 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 636037 T2162 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 636040 T2162 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 636042 T2162 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 636045 T2162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 636046 T2162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 636046 T2162 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 636047 T2162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 636047 T2162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 636048 T2162 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 636048 T2162 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371609854752/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/
[junit4:junit4] 2> 636048 T2162 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2625e90a
[junit4:junit4] 2> 636049 T2162 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 636049 T2162 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1
[junit4:junit4] 2> 636049 T2162 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index/
[junit4:junit4] 2> 636050 T2162 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 636050 T2162 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index
[junit4:junit4] 2> 636053 T2162 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 636054 T2162 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 636056 T2162 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 636056 T2162 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 636056 T2162 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 636057 T2162 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 636057 T2162 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 636058 T2162 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 636058 T2162 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 636058 T2162 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 636059 T2162 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 636061 T2162 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 636064 T2162 oass.SolrIndexSearcher.<init> Opening Searcher@14a75d7a main
[junit4:junit4] 2> 636064 T2162 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 636065 T2162 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 636068 T2163 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14a75d7a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 636070 T2162 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 636070 T2162 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56867 collection:collection1 shard:shard1
[junit4:junit4] 2> 636071 T2162 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 636076 T2162 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 636078 T2162 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 636078 T2162 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 636078 T2162 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 636078 T2162 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 636079 T2162 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/ has no replicas
[junit4:junit4] 2> 636079 T2162 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 636079 T2162 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 636312 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 636317 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 636317 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 636317 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 636334 T2162 oasc.ZkController.register We are http://127.0.0.1:56867/collection1/ and leader is http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 636334 T2162 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56867
[junit4:junit4] 2> 636334 T2162 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 636335 T2162 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 636335 T2162 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 636337 T2162 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 636338 T2119 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 636339 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 636339 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 636405 T2119 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 636408 T2119 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35505
[junit4:junit4] 2> 636409 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 636409 T2119 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 636410 T2119 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766
[junit4:junit4] 2> 636410 T2119 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/solr.xml
[junit4:junit4] 2> 636411 T2119 oasc.CoreContainer.<init> New CoreContainer 736752698
[junit4:junit4] 2> 636411 T2119 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/'
[junit4:junit4] 2> 636411 T2119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/'
[junit4:junit4] 2> 636470 T2119 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 636471 T2119 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 636472 T2119 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 636472 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 636473 T2119 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 636473 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 636473 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 636474 T2119 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 636474 T2119 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 636474 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 636477 T2119 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 636478 T2119 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50920/solr
[junit4:junit4] 2> 636478 T2119 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 636479 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 636481 T2175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31862bb3 name:ZooKeeperConnection Watcher:127.0.0.1:50920 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 636482 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 636485 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 636489 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 636490 T2177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f96f335 name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 636490 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 636494 T2119 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 637496 T2119 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35505_
[junit4:junit4] 2> 637498 T2119 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35505_
[junit4:junit4] 2> 637499 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 637499 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 637500 T2141 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 637501 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 637501 T2177 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 637502 T2161 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 637502 T2148 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 637506 T2178 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 637507 T2178 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 637824 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 637825 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56867_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56867"}
[junit4:junit4] 2> 637828 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35505_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35505"}
[junit4:junit4] 2> 637828 T2142 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 637828 T2142 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 637831 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 637831 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 637831 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 637831 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 638508 T2178 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/collection1
[junit4:junit4] 2> 638508 T2178 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 638509 T2178 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 638509 T2178 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 638510 T2178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/collection1/'
[junit4:junit4] 2> 638511 T2178 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/collection1/lib/README' to classloader
[junit4:junit4] 2> 638511 T2178 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 638544 T2178 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 638580 T2178 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 638581 T2178 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 638584 T2178 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 638964 T2178 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 638969 T2178 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 638971 T2178 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 638981 T2178 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 638984 T2178 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 638986 T2178 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 638987 T2178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 638988 T2178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 638988 T2178 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 638989 T2178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 638989 T2178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 638989 T2178 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 638989 T2178 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371609857766/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/
[junit4:junit4] 2> 638990 T2178 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2625e90a
[junit4:junit4] 2> 638990 T2178 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 638990 T2178 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2
[junit4:junit4] 2> 638991 T2178 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index/
[junit4:junit4] 2> 638991 T2178 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 638991 T2178 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index
[junit4:junit4] 2> 638994 T2178 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 638995 T2178 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 638996 T2178 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 638997 T2178 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 638997 T2178 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 638998 T2178 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 638998 T2178 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 638998 T2178 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 638999 T2178 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 638999 T2178 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 638999 T2178 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 639001 T2178 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 639010 T2178 oass.SolrIndexSearcher.<init> Opening Searcher@6a20e393 main
[junit4:junit4] 2> 639011 T2178 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 639011 T2178 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 639016 T2179 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a20e393 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 639018 T2178 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 639018 T2178 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35505 collection:collection1 shard:shard1
[junit4:junit4] 2> 639021 T2178 oasc.ZkController.register We are http://127.0.0.1:35505/collection1/ and leader is http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 639021 T2178 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35505
[junit4:junit4] 2> 639021 T2178 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 639022 T2178 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1745 name=collection1 org.apache.solr.core.SolrCore@424db64f url=http://127.0.0.1:35505/collection1 node=127.0.0.1:35505_ C1745_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:35505_, base_url=http://127.0.0.1:35505}
[junit4:junit4] 2> 639022 T2180 C1745 P35505 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 639023 T2178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 639023 T2180 C1745 P35505 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 639024 T2180 C1745 P35505 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 639024 T2180 C1745 P35505 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 639024 T2119 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 639025 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 639025 T2180 C1745 P35505 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 639025 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 639029 T2156 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 639098 T2119 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 639100 T2119 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47774
[junit4:junit4] 2> 639101 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 639101 T2119 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 639101 T2119 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452
[junit4:junit4] 2> 639102 T2119 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/solr.xml
[junit4:junit4] 2> 639102 T2119 oasc.CoreContainer.<init> New CoreContainer 1625116462
[junit4:junit4] 2> 639103 T2119 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/'
[junit4:junit4] 2> 639103 T2119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/'
[junit4:junit4] 2> 639166 T2119 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 639167 T2119 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 639167 T2119 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 639168 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 639168 T2119 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 639169 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 639169 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 639170 T2119 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 639170 T2119 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 639171 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 639175 T2119 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 639176 T2119 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50920/solr
[junit4:junit4] 2> 639176 T2119 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 639177 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 639180 T2192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72b4c193 name:ZooKeeperConnection Watcher:127.0.0.1:50920 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 639181 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 639183 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 639187 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 639189 T2194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e31c91f name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 639189 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 639192 T2119 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 639336 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 639336 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35505_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35505"}
[junit4:junit4] 2> 639339 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 639339 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 639339 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 639340 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 639340 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 640030 T2156 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 640030 T2156 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:35505_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 640195 T2119 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47774_
[junit4:junit4] 2> 640197 T2119 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47774_
[junit4:junit4] 2> 640199 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 640199 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 640199 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 640199 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 640199 T2141 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 640200 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 640200 T2161 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 640201 T2194 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 640201 T2177 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 640201 T2148 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 640206 T2195 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 640206 T2195 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 640843 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 640844 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47774_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47774"}
[junit4:junit4] 2> 640844 T2142 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 640844 T2142 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 640909 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 640909 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 640909 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 640909 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 640909 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 641208 T2195 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/collection1
[junit4:junit4] 2> 641208 T2195 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 641209 T2195 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 641209 T2195 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 641210 T2195 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/collection1/'
[junit4:junit4] 2> 641210 T2195 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/collection1/lib/README' to classloader
[junit4:junit4] 2> 641211 T2195 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 641242 T2195 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 641276 T2195 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 641277 T2195 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 641282 T2195 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 641757 T2195 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 641766 T2195 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 641770 T2195 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 641783 T2195 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 641788 T2195 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 641792 T2195 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 641794 T2195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 641794 T2195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 641795 T2195 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 641796 T2195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 641796 T2195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 641796 T2195 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 641797 T2195 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371609860452/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/
[junit4:junit4] 2> 641797 T2195 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2625e90a
[junit4:junit4] 2> 641798 T2195 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 641798 T2195 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3
[junit4:junit4] 2> 641799 T2195 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index/
[junit4:junit4] 2> 641799 T2195 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 641799 T2195 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index
[junit4:junit4] 2> 641804 T2195 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 641805 T2195 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 641808 T2195 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 641808 T2195 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 641809 T2195 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 641810 T2195 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 641811 T2195 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 641811 T2195 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 641811 T2195 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 641812 T2195 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 641813 T2195 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 641816 T2195 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 641821 T2195 oass.SolrIndexSearcher.<init> Opening Searcher@31d27dc3 main
[junit4:junit4] 2> 641822 T2195 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 641822 T2195 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 641827 T2196 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31d27dc3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 641829 T2195 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 641830 T2195 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47774 collection:collection1 shard:shard1
[junit4:junit4] 2> 641832 T2195 oasc.ZkController.register We are http://127.0.0.1:47774/collection1/ and leader is http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 641833 T2195 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47774
[junit4:junit4] 2> 641833 T2195 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 641833 T2195 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1746 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:47774/collection1 node=127.0.0.1:47774_ C1746_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47774_, base_url=http://127.0.0.1:47774}
[junit4:junit4] 2> 641834 T2197 C1746 P47774 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 641834 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 641834 T2197 C1746 P47774 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 641835 T2197 C1746 P47774 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 641835 T2197 C1746 P47774 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 641835 T2119 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 641836 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 641836 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 641837 T2197 C1746 P47774 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 641841 T2157 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 641939 T2119 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 641942 T2119 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54710
[junit4:junit4] 2> 641943 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 641943 T2119 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 641943 T2119 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264
[junit4:junit4] 2> 641944 T2119 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/solr.xml
[junit4:junit4] 2> 641944 T2119 oasc.CoreContainer.<init> New CoreContainer 1590304874
[junit4:junit4] 2> 641944 T2119 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/'
[junit4:junit4] 2> 641945 T2119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/'
[junit4:junit4] 2> 641992 T2119 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 641993 T2119 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 641993 T2119 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 641993 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 641994 T2119 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 641994 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 641994 T2119 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 641994 T2119 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 641995 T2119 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 641995 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 641998 T2119 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 641998 T2119 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50920/solr
[junit4:junit4] 2> 641999 T2119 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 641999 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 642001 T2209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a9804 name:ZooKeeperConnection Watcher:127.0.0.1:50920 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 642002 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 642004 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 642006 T2119 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 642016 T2211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b3f649d name:ZooKeeperConnection Watcher:127.0.0.1:50920/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 642017 T2119 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 642020 T2119 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1747 name=collection1 org.apache.solr.core.SolrCore@424db64f url=http://127.0.0.1:35505/collection1 node=127.0.0.1:35505_ C1747_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35505_, base_url=http://127.0.0.1:35505}
[junit4:junit4] 2> 642031 T2180 C1747 P35505 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:56867/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 642031 T2180 C1747 P35505 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35505 START replicas=[http://127.0.0.1:56867/collection1/] nUpdates=100
[junit4:junit4] 2> 642032 T2180 C1747 P35505 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 642032 T2180 C1747 P35505 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 642032 T2180 C1747 P35505 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 642033 T2180 C1747 P35505 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 642033 T2180 C1747 P35505 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 642033 T2180 C1747 P35505 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:56867/collection1/. core=collection1
[junit4:junit4] 2> 642033 T2180 C1747 P35505 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1748 name=collection1 org.apache.solr.core.SolrCore@38be845b url=http://127.0.0.1:56867/collection1 node=127.0.0.1:56867_ C1748_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56867_, base_url=http://127.0.0.1:56867, leader=true}
[junit4:junit4] 2> 642039 T2154 C1748 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 642041 T2155 C1748 P56867 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 642044 T2155 C1748 P56867 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 642044 T2155 C1748 P56867 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 642049 T2155 C1748 P56867 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 642049 T2155 C1748 P56867 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 642050 T2155 C1748 P56867 oass.SolrIndexSearcher.<init> Opening Searcher@302386bd realtime
[junit4:junit4] 2> 642050 T2155 C1748 P56867 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 642051 T2155 C1748 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 642052 T2180 C1747 P35505 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 642052 T2180 C1747 P35505 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 642053 T2156 C1748 P56867 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 642054 T2156 C1748 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 642054 T2180 C1747 P35505 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 642054 T2180 C1747 P35505 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 642055 T2180 C1747 P35505 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 642057 T2155 C1748 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 642057 T2180 C1747 P35505 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 642058 T2180 C1747 P35505 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index.20130619044423482
[junit4:junit4] 2> 642058 T2180 C1747 P35505 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index.20130619044423482 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cc514b; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 642061 T2154 C1748 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 642062 T2180 C1747 P35505 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 642069 T2180 C1747 P35505 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 642070 T2180 C1747 P35505 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 642072 T2180 C1747 P35505 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 642073 T2180 C1747 P35505 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 642073 T2180 C1747 P35505 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 642073 T2180 C1747 P35505 oass.SolrIndexSearcher.<init> Opening Searcher@608498f1 main
[junit4:junit4] 2> 642074 T2179 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@608498f1 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 642075 T2180 C1747 P35505 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index.20130619044423482 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index.20130619044423482;done=true>>]
[junit4:junit4] 2> 642075 T2180 C1747 P35505 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index.20130619044423482
[junit4:junit4] 2> 642075 T2180 C1747 P35505 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index.20130619044423482
[junit4:junit4] 2> 642076 T2180 C1747 P35505 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 642076 T2180 C1747 P35505 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 642076 T2180 C1747 P35505 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 642076 T2180 C1747 P35505 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 642078 T2180 C1747 P35505 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 642415 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 642416 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47774_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47774"}
[junit4:junit4] 2> 642418 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35505_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35505"}
[junit4:junit4] 2> 642439 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 642439 T2211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 642439 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 642439 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 642439 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 642439 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 642842 T2157 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 642843 T2157 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:47774_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 643023 T2119 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54710_
[junit4:junit4] 2> 643025 T2119 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54710_
[junit4:junit4] 2> 643027 T2211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 643028 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 643028 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 643028 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 643027 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 643030 T2141 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 643030 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 643031 T2177 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 643031 T2161 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 643031 T2194 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 643031 T2148 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 643032 T2211 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 643038 T2214 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 643038 T2214 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 643946 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 643947 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54710_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54710"}
[junit4:junit4] 2> 643947 T2142 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 643947 T2142 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 643950 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 643950 T2211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 643950 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 643950 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 643950 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 643950 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 644040 T2214 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/collection1
[junit4:junit4] 2> 644040 T2214 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 644041 T2214 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 644041 T2214 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 644042 T2214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/collection1/'
[junit4:junit4] 2> 644043 T2214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/collection1/lib/README' to classloader
[junit4:junit4] 2> 644044 T2214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 644094 T2214 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 644135 T2214 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 644142 T2214 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 644146 T2214 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 644572 T2214 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 644577 T2214 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 644580 T2214 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 644590 T2214 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 644593 T2214 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 644596 T2214 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 644597 T2214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 644597 T2214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 644598 T2214 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 644598 T2214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 644599 T2214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 644599 T2214 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 644599 T2214 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371609863264/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/
[junit4:junit4] 2> 644599 T2214 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2625e90a
[junit4:junit4] 2> 644600 T2214 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 644600 T2214 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4
[junit4:junit4] 2> 644600 T2214 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index/
[junit4:junit4] 2> 644601 T2214 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 644601 T2214 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index
[junit4:junit4] 2> 644604 T2214 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 644605 T2214 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 644607 T2214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 644607 T2214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 644608 T2214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 644608 T2214 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 644609 T2214 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 644609 T2214 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 644609 T2214 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 644610 T2214 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 644610 T2214 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 644613 T2214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 644616 T2214 oass.SolrIndexSearcher.<init> Opening Searcher@510febdf main
[junit4:junit4] 2> 644617 T2214 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 644618 T2214 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 644622 T2215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@510febdf main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 644624 T2214 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 644624 T2214 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54710 collection:collection1 shard:shard1
[junit4:junit4] 2> 644627 T2214 oasc.ZkController.register We are http://127.0.0.1:54710/collection1/ and leader is http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 644627 T2214 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54710
[junit4:junit4] 2> 644627 T2214 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 644628 T2214 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1749 name=collection1 org.apache.solr.core.SolrCore@54c4830f url=http://127.0.0.1:54710/collection1 node=127.0.0.1:54710_ C1749_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:54710_, base_url=http://127.0.0.1:54710}
[junit4:junit4] 2> 644628 T2216 C1749 P54710 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 644628 T2214 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 644629 T2216 C1749 P54710 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 644630 T2216 C1749 P54710 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 644630 T2216 C1749 P54710 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 644630 T2119 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 644631 T2119 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 644631 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 644635 T2216 C1749 P54710 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 644635 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 644636 T2119 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 644637 T2119 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 644638 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 644639 T2157 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C1750 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:47774/collection1 node=127.0.0.1:47774_ C1750_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:47774_, base_url=http://127.0.0.1:47774}
[junit4:junit4] 2> 644843 T2197 C1750 P47774 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:56867/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 644844 T2197 C1750 P47774 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47774 START replicas=[http://127.0.0.1:56867/collection1/] nUpdates=100
[junit4:junit4] 2> 644844 T2197 C1750 P47774 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 644845 T2197 C1750 P47774 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 644845 T2197 C1750 P47774 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 644845 T2197 C1750 P47774 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 644845 T2197 C1750 P47774 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 644845 T2197 C1750 P47774 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:56867/collection1/. core=collection1
[junit4:junit4] 2> 644845 T2197 C1750 P47774 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1751 name=collection1 org.apache.solr.core.SolrCore@38be845b url=http://127.0.0.1:56867/collection1 node=127.0.0.1:56867_ C1751_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56867_, base_url=http://127.0.0.1:56867, leader=true}
[junit4:junit4] 2> 644847 T2154 C1751 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 644849 T2156 C1751 P56867 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 644859 T2156 C1751 P56867 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 644859 T2156 C1751 P56867 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 644860 T2156 C1751 P56867 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 644860 T2156 C1751 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 644861 T2197 C1750 P47774 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 644861 T2197 C1750 P47774 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 644864 T2154 C1751 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 644865 T2197 C1750 P47774 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 644865 T2197 C1750 P47774 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 644865 T2197 C1750 P47774 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 644867 T2156 C1751 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 644868 T2197 C1750 P47774 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 644868 T2197 C1750 P47774 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index.20130619044426292
[junit4:junit4] 2> 644869 T2197 C1750 P47774 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index.20130619044426292 lockFactory=org.apache.lucene.store.NativeFSLockFactory@66e003e1; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> ASYNC NEW_CORE C1752 name=collection1 org.apache.solr.core.SolrCore@38be845b url=http://127.0.0.1:56867/collection1 node=127.0.0.1:56867_ C1752_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56867_, base_url=http://127.0.0.1:56867, leader=true}
[junit4:junit4] 2> 644877 T2154 C1752 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C1753 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:47774/collection1 node=127.0.0.1:47774_ C1753_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:47774_, base_url=http://127.0.0.1:47774}
[junit4:junit4] 2> 644879 T2197 C1753 P47774 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 644881 T2197 C1753 P47774 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 644881 T2197 C1753 P47774 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 644883 T2197 C1753 P47774 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 644884 T2197 C1753 P47774 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 644884 T2197 C1753 P47774 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 644885 T2197 C1753 P47774 oass.SolrIndexSearcher.<init> Opening Searcher@7f5001ba main
[junit4:junit4] 2> 644886 T2196 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f5001ba main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 644886 T2197 C1753 P47774 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index.20130619044426292 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index.20130619044426292;done=true>>]
[junit4:junit4] 2> 644887 T2197 C1753 P47774 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index.20130619044426292
[junit4:junit4] 2> 644887 T2197 C1753 P47774 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index.20130619044426292
[junit4:junit4] 2> 644887 T2197 C1753 P47774 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 644888 T2197 C1753 P47774 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 644888 T2197 C1753 P47774 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 644888 T2197 C1753 P47774 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 644909 T2197 C1753 P47774 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 645454 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 645455 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54710_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54710"}
[junit4:junit4] 2> 645459 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47774_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47774"}
[junit4:junit4] 2> 645462 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 645462 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 645462 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 645462 T2211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 645462 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 645462 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 645640 T2157 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 645640 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 645640 T2157 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:54710_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 646642 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1754 name=collection1 org.apache.solr.core.SolrCore@54c4830f url=http://127.0.0.1:54710/collection1 node=127.0.0.1:54710_ C1754_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:54710_, base_url=http://127.0.0.1:54710}
[junit4:junit4] 2> 647642 T2216 C1754 P54710 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:56867/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 647642 T2216 C1754 P54710 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54710 START replicas=[http://127.0.0.1:56867/collection1/] nUpdates=100
[junit4:junit4] 2> 647642 T2216 C1754 P54710 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 647642 T2216 C1754 P54710 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 647643 T2216 C1754 P54710 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 647643 T2216 C1754 P54710 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 647643 T2216 C1754 P54710 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 647643 T2216 C1754 P54710 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:56867/collection1/. core=collection1
[junit4:junit4] 2> 647643 T2216 C1754 P54710 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 647643 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 647644 T2157 C1752 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 647647 T2155 C1752 P56867 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 647719 T2155 C1752 P56867 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 647719 T2155 C1752 P56867 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 647720 T2155 C1752 P56867 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 647720 T2155 C1752 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 73
[junit4:junit4] 2> 647721 T2216 C1754 P54710 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 647721 T2216 C1754 P54710 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 647723 T2154 C1752 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 647724 T2216 C1754 P54710 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 647724 T2216 C1754 P54710 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 647724 T2216 C1754 P54710 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 647727 T2155 C1752 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 647728 T2216 C1754 P54710 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 647729 T2216 C1754 P54710 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index.20130619044429152
[junit4:junit4] 2> 647730 T2216 C1754 P54710 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index.20130619044429152 lockFactory=org.apache.lucene.store.NativeFSLockFactory@52f16e9b; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 647732 T2156 C1752 P56867 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 647734 T2216 C1754 P54710 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 647736 T2216 C1754 P54710 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 647736 T2216 C1754 P54710 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 647738 T2216 C1754 P54710 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 647739 T2216 C1754 P54710 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 647739 T2216 C1754 P54710 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 647739 T2216 C1754 P54710 oass.SolrIndexSearcher.<init> Opening Searcher@6e2b6d92 main
[junit4:junit4] 2> 647740 T2215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e2b6d92 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 647741 T2216 C1754 P54710 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index.20130619044429152 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index.20130619044429152;done=true>>]
[junit4:junit4] 2> 647741 T2216 C1754 P54710 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index.20130619044429152
[junit4:junit4] 2> 647741 T2216 C1754 P54710 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index.20130619044429152
[junit4:junit4] 2> 647741 T2216 C1754 P54710 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 647741 T2216 C1754 P54710 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 647742 T2216 C1754 P54710 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 647742 T2216 C1754 P54710 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 647743 T2216 C1754 P54710 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 648468 T2142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 648469 T2142 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54710_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54710"}
[junit4:junit4] 2> 648511 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 648511 T2211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 648511 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 648511 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 648511 T2161 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 648511 T2177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 648645 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 648646 T2119 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C1755 name=collection1 org.apache.solr.core.SolrCore@6c519115 url=http://127.0.0.1:40028/collection1 node=127.0.0.1:40028_ C1755_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:40028_, base_url=http://127.0.0.1:40028, leader=true}
[junit4:junit4] 2> 648654 T2137 C1755 P40028 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 648657 T2137 C1755 P40028 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 648658 T2137 C1755 P40028 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 648679 T2137 C1755 P40028 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 648679 T2137 C1755 P40028 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 648680 T2137 C1755 P40028 oass.SolrIndexSearcher.<init> Opening Searcher@769c523c main
[junit4:junit4] 2> 648681 T2137 C1755 P40028 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 648681 T2145 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@769c523c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 648682 T2137 C1755 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 28
[junit4:junit4] 2> 648685 T2157 C1752 P56867 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:56867/collection1/, StdNode: http://127.0.0.1:35505/collection1/, StdNode: http://127.0.0.1:47774/collection1/, StdNode: http://127.0.0.1:54710/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 648688 T2154 C1752 P56867 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C1753_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:47774_, base_url=http://127.0.0.1:47774}
[junit4:junit4] 2> 648693 T2189 C1753 P47774 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C1754_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54710_, base_url=http://127.0.0.1:54710}
[junit4:junit4] 2> 648695 T2204 C1754 P54710 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C1756 name=collection1 org.apache.solr.core.SolrCore@424db64f url=http://127.0.0.1:35505/collection1 node=127.0.0.1:35505_ C1756_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35505_, base_url=http://127.0.0.1:35505}
[junit4:junit4] 2> 648697 T2167 C1756 P35505 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 648709 T2154 C1752 P56867 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 648710 T2154 C1752 P56867 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 648710 T2154 C1752 P56867 oass.SolrIndexSearcher.<init> Opening Searcher@3e183ec4 main
[junit4:junit4] 2> 648711 T2154 C1752 P56867 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 648711 T2163 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3e183ec4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 648712 T2154 C1752 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4] 2> 648739 T2189 C1753 P47774 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 648740 T2189 C1753 P47774 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 648739 T2167 C1756 P35505 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 648740 T2204 C1754 P54710 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 648741 T2189 C1753 P47774 oass.SolrIndexSearcher.<init> Opening Searcher@79c02dba main
[junit4:junit4] 2> 648741 T2167 C1756 P35505 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 648742 T2189 C1753 P47774 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 648741 T2204 C1754 P54710 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 648742 T2167 C1756 P35505 oass.SolrIndexSearcher.<init> Opening Searcher@1e949220 main
[junit4:junit4] 2> 648742 T2196 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c02dba main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 648743 T2204 C1754 P54710 oass.SolrIndexSearcher.<init> Opening Searcher@4a18ac6c main
[junit4:junit4] 2> 648743 T2167 C1756 P35505 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 648744 T2179 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e949220 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 648744 T2204 C1754 P54710 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 648744 T2189 C1753 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 51
[junit4:junit4] 2> 648744 T2167 C1756 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 47
[junit4:junit4] 2> 648744 T2215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a18ac6c main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 648746 T2204 C1754 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 51
[junit4:junit4] 2> 648746 T2157 C1752 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 61
[junit4:junit4] 2> 648747 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 648749 T2155 C1752 P56867 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 648753 T2171 C1756 P35505 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 648755 T2188 C1753 P47774 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 648757 T2206 C1754 P54710 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 650763 T2136 C1755 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438237193336258560)} 0 3
[junit4:junit4] 2> ASYNC NEW_CORE C1757 name=collection1 org.apache.solr.core.SolrCore@54c4830f url=http://127.0.0.1:54710/collection1 node=127.0.0.1:54710_ C1757_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54710_, base_url=http://127.0.0.1:54710}
[junit4:junit4] 2> 650776 T2205 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438237193347792896&update.from=http://127.0.0.1:56867/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438237193347792896)} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C1758 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:47774/collection1 node=127.0.0.1:47774_ C1758_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:47774_, base_url=http://127.0.0.1:47774}
[junit4:junit4] 2> 650778 T2189 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438237193347792896&update.from=http://127.0.0.1:56867/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438237193347792896)} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C1759 name=collection1 org.apache.solr.core.SolrCore@424db64f url=http://127.0.0.1:35505/collection1 node=127.0.0.1:35505_ C1759_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35505_, base_url=http://127.0.0.1:35505}
[junit4:junit4] 2> 650776 T2167 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438237193347792896&update.from=http://127.0.0.1:56867/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438237193347792896)} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C1760 name=collection1 org.apache.solr.core.SolrCore@38be845b url=http://127.0.0.1:56867/collection1 node=127.0.0.1:56867_ C1760_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56867_, base_url=http://127.0.0.1:56867, leader=true}
[junit4:junit4] 2> 650779 T2157 C1760 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438237193347792896)} 0 7
[junit4:junit4] 2> ASYNC NEW_CORE C1761 name=collection1 org.apache.solr.core.SolrCore@6c519115 url=http://127.0.0.1:40028/collection1 node=127.0.0.1:40028_ C1761_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:40028_, base_url=http://127.0.0.1:40028, leader=true}
[junit4:junit4] 2> 650784 T2137 C1761 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438237193358278656)]} 0 2
[junit4:junit4] 2> 650795 T2172 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438237193365618688)]} 0 2
[junit4:junit4] 2> 650795 T2207 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438237193365618688)]} 0 2
[junit4:junit4] 2> 650795 T2188 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438237193365618688)]} 0 2
[junit4:junit4] 2> 650797 T2155 C1760 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35505/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438237193365618688)]} 0 8
[junit4:junit4] 2> 650798 T2171 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 10
[junit4:junit4] 2> 650802 T2134 C1761 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438237193379250176)]} 0 1
[junit4:junit4] 2> 650810 T2173 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438237193384493056)]} 0 0
[junit4:junit4] 2> 650811 T2190 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438237193384493056)]} 0 1
[junit4:junit4] 2> 650810 T2205 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438237193384493056)]} 0 0
[junit4:junit4] 2> 650812 T2154 C1760 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54710/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438237193384493056)]} 0 5
[junit4:junit4] 2> 650813 T2206 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 9
[junit4:junit4] 2> 650816 T2135 C1761 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438237193392881664)]} 0 1
[junit4:junit4] 2> 650821 T2157 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:35505/collection1/]
[junit4:junit4] 2> 650822 T2157 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35505/collection1/ against:[http://127.0.0.1:35505/collection1/] result:true
[junit4:junit4] 2> 650822 T2157 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:47774/collection1/ against:[http://127.0.0.1:35505/collection1/] result:false
[junit4:junit4] 2> 650822 T2157 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:54710/collection1/ against:[http://127.0.0.1:35505/collection1/] result:false
[junit4:junit4] 2> 650826 T2187 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[2 (1438237193401270272)]} 0 1
[junit4:junit4] 2> 650826 T2204 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[2 (1438237193401270272)]} 0 1
[junit4:junit4] 2> 650827 T2157 C1760 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54710/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[2 (1438237193401270272)]} 0 6
[junit4:junit4] 2> 650827 T2207 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[2]} 0 8
[junit4:junit4] 2> 650830 T2136 C1761 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438237193408610304)]} 0 0
[junit4:junit4] 2> 650834 T2155 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:35505/collection1/]
[junit4:junit4] 2> 650834 T2155 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35505/collection1/ against:[http://127.0.0.1:35505/collection1/] result:true
[junit4:junit4] 2> 650835 T2155 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:47774/collection1/ against:[http://127.0.0.1:35505/collection1/] result:false
[junit4:junit4] 2> 650835 T2155 C1760 P56867 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:54710/collection1/ against:[http://127.0.0.1:35505/collection1/] result:false
[junit4:junit4] 2> 650838 T2205 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[3 (1438237193413853184)]} 0 0
[junit4:junit4] 2> 650838 T2188 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56867/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[3 (1438237193413853184)]} 0 0
[junit4:junit4] 2> 650839 T2155 C1760 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:47774/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&wt=javabin&version=2} {add=[3 (1438237193413853184)]} 0 5
[junit4:junit4] 2> 650840 T2189 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:35505/collection1/&test.distrib.skip.servers=http://127.0.0.1:47774/collection1/&wt=javabin&version=2} {add=[3]} 0 8
[junit4:junit4] 2> 650842 T2137 C1761 P40028 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 650909 T2137 C1761 P40028 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 650909 T2137 C1761 P40028 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 650911 T2137 C1761 P40028 oass.SolrIndexSearcher.<init> Opening Searcher@3ac9c7bf main
[junit4:junit4] 2> 650912 T2137 C1761 P40028 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 650913 T2145 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ac9c7bf main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 650913 T2137 C1761 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 71
[junit4:junit4] 2> 650916 T2167 C1759 P35505 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:56867/collection1/, StdNode: http://127.0.0.1:35505/collection1/, StdNode: http://127.0.0.1:47774/collection1/, StdNode: http://127.0.0.1:54710/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 650918 T2187 C1758 P47774 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 650919 T2171 C1759 P35505 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 650919 T2156 C1760 P56867 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 650928 T2204 C1757 P54710 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 650959 T2187 C1758 P47774 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 650959 T2171 C1759 P35505 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@62d6662d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 650960 T2171 C1759 P35505 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 650960 T2187 C1758 P47774 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 650962 T2187 C1758 P47774 oass.SolrIndexSearcher.<init> Opening Searcher@2cd1ab9f main
[junit4:junit4] 2> 650962 T2171 C1759 P35505 oass.SolrIndexSearcher.<init> Opening Searcher@2cd1ab9f main
[junit4:junit4] 2> 650963 T2171 C1759 P35505 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 650964 T2179 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 650964 T2187 C1758 P47774 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 650964 T2196 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2cd1ab9f main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 650965 T2179 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2cd1ab9f main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 650965 T2187 C1758 P47774 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 47
[junit4:junit4] 2> 650965 T2171 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 650969 T2204 C1757 P54710 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e502258; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 650969 T2204 C1757 P54710 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 650970 T2156 C1760 P56867 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f162861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 650970 T2156 C1760 P56867 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 650972 T2204 C1757 P54710 oass.SolrIndexSearcher.<init> Opening Searcher@3b39e234 main
[junit4:junit4] 2> 650973 T2156 C1760 P56867 oass.SolrIndexSearcher.<init> Opening Searcher@d4c8158 main
[junit4:junit4] 2> 650973 T2156 C1760 P56867 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 650974 T2215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b39e234 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 650975 T2163 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d4c8158 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 650975 T2156 C1760 P56867 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 57
[junit4:junit4] 2> 650973 T2204 C1757 P54710 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 650976 T2204 C1757 P54710 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 48
[junit4:junit4] 2> 650977 T2167 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 61
[junit4:junit4] 2> 650977 T2119 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 650978 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 650979 T2119 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 650982 T2154 C1760 P56867 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 650985 T2173 C1759 P35505 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=1
[junit4:junit4] 2> 650988 T2188 C1758 P47774 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 650991 T2207 C1757 P54710 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 650992 T2119 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 650995 T2189 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 650996 T2189 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 650999 T2157 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 651000 T2157 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56867/collection1/
[junit4:junit4] 2> 651001 T2157 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56867 START replicas=[http://127.0.0.1:35505/collection1/, http://127.0.0.1:47774/collection1/, http://127.0.0.1:54710/collection1/] nUpdates=100
[junit4:junit4] 2> 651003 T2190 C1758 P47774 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 651003 T2205 C1757 P54710 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 651004 T2157 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56867 Received 5 versions from 127.0.0.1:47774/collection1/
[junit4:junit4] 2> 651004 T2171 C1759 P35505 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 651004 T2157 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56867 Our versions are newer. ourLowThreshold=1438237193347792896 otherHigh=1438237193401270272
[junit4:junit4] 2> 651005 T2157 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56867 Received 5 versions from 127.0.0.1:54710/collection1/
[junit4:junit4] 2> 651005 T2157 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56867 Our versions are newer. ourLowThreshold=1438237193347792896 otherHigh=1438237193401270272
[junit4:junit4] 2> 651005 T2157 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56867 Received 3 versions from 127.0.0.1:35505/collection1/
[junit4:junit4] 2> 651006 T2157 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56867 Our versions are newer. ourLowThreshold=1438237193347792896 otherHigh=1438237193384493056
[junit4:junit4] 2> 651006 T2157 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56867 DONE. sync succeeded
[junit4:junit4] 2> 651006 T2157 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 651006 T2157 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/: try and ask http://127.0.0.1:35505/collection1/ to sync
[junit4:junit4] 2> 651007 T2157 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/: try and ask http://127.0.0.1:47774/collection1/ to sync
[junit4:junit4] 2> 651007 T2157 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/: try and ask http://127.0.0.1:54710/collection1/ to sync
[junit4:junit4] 2> 651009 T2173 C1759 P35505 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35505 START replicas=[http://127.0.0.1:56867/collection1/] nUpdates=100
[junit4:junit4] 2> 651009 T2188 C1758 P47774 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47774 START replicas=[http://127.0.0.1:56867/collection1/] nUpdates=100
[junit4:junit4] 2> 651012 T2204 C1757 P54710 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54710 START replicas=[http://127.0.0.1:56867/collection1/] nUpdates=100
[junit4:junit4] 2> 651013 T2154 C1760 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 651013 T2188 C1758 P47774 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:47774 Received 5 versions from 127.0.0.1:56867/collection1/
[junit4:junit4] 2> 651014 T2188 C1758 P47774 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:47774 Our versions are newer. ourLowThreshold=1438237193347792896 otherHigh=1438237193401270272
[junit4:junit4] 2> 651014 T2188 C1758 P47774 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47774 DONE. sync succeeded
[junit4:junit4] 2> 651015 T2188 C1758 P47774 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:56867/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 651015 T2157 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/: sync completed with http://127.0.0.1:47774/collection1/
[junit4:junit4] 2> 651017 T2156 C1760 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=5
[junit4:junit4] 2> 651018 T2173 C1759 P35505 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35505 Received 5 versions from 127.0.0.1:56867/collection1/
[junit4:junit4] 2> 651018 T2173 C1759 P35505 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:35505 Requesting updates from 127.0.0.1:56867/collection1/n=2 versions=[1438237193413853184, 1438237193401270272]
[junit4:junit4] 2> 651018 T2155 C1760 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 651019 T2204 C1757 P54710 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54710 Received 5 versions from 127.0.0.1:56867/collection1/
[junit4:junit4] 2> 651019 T2204 C1757 P54710 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54710 Our versions are newer. ourLowThreshold=1438237193347792896 otherHigh=1438237193401270272
[junit4:junit4] 2> 651020 T2204 C1757 P54710 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54710 DONE. sync succeeded
[junit4:junit4] 2> 651020 T2204 C1757 P54710 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:56867/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 651020 T2157 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/: sync completed with http://127.0.0.1:54710/collection1/
[junit4:junit4] 2> 651021 T2154 C1760 P56867 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1438237193413853184,1438237193401270272&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 651025 T2173 C1759 P35505 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438237193401270272), 3 (1438237193413853184)]} 0 3
[junit4:junit4] 2> 651025 T2173 C1759 P35505 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35505 DONE. sync succeeded
[junit4:junit4] 2> 651026 T2173 C1759 P35505 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:56867/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=18
[junit4:junit4] 2> 651027 T2157 oasc.SyncStrategy.syncToMe http://127.0.0.1:56867/collection1/: sync completed with http://127.0.0.1:35505/collection1/
[junit4:junit4] 2> 651027 T2157 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=28
[junit4:junit4] 2> 651028 T2189 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=33
[junit4:junit4] 2> 651028 T2119 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 651029 T2119 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 651030 T2119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 651030 T2119 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 651032 T2135 C1761 P40028 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 651079 T2135 C1761 P40028 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@617f6bd1; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 651079 T2135 C1761 P40028 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 651080 T2135 C1761 P40028 oass.SolrIndexSearcher.<init> Opening Searcher@28384c04 main
[junit4:junit4] 2> 651081 T2135 C1761 P40028 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 651082 T2145 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28384c04 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 651082 T2135 C1761 P40028 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 50
[junit4:junit4] 2> 651085 T2207 C1757 P54710 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:56867/collection1/, StdNode: http://127.0.0.1:35505/collection1/, StdNode: http://127.0.0.1:47774/collection1/, StdNode: http://127.0.0.1:54710/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 651088 T2205 C1757 P54710 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 651088 T2187 C1758 P47774 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 651088 T2156 C1760 P56867 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 651088 T2171 C1759 P35505 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 651109 T2187 C1758 P47774 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371609851229/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b1e5a7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5
[...truncated too long message...]
"shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:47774_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47774",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "5":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:54710_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:54710"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:40028_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40028",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40028_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:56867_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:47774_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:54710_ (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89889823210340362-4-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89889823210340366-2-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89889823210340364-5-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:47774_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47774"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89889823210340355-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:40028_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40028"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89889823210340366-127.0.0.1:56867_-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89889823210340355-127.0.0.1:40028_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89889823210340362-127.0.0.1:47774_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89889823210340364-127.0.0.1:54710_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89889823210340355-127.0.0.1:40028_-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=82C33D59EEE16665 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=be_BY -Dtests.timezone=Europe/Warsaw -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 102s J1 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:56867/collection1lastClient and got 258 from http://127.0.0.1:47774/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([82C33D59EEE16665:325B34199BE0659]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 731968 T2119 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 102171 T2118 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=be_BY, timezone=Europe/Warsaw
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=185899176,total=311046144
[junit4:junit4] 2> NOTE: All tests run in this JVM: [DisMaxRequestHandlerTest, IndexBasedSpellCheckerTest, SampleTest, SpellPossibilityIteratorTest, TestSolrQueryParserResource, TestZkChroot, TestFastOutputStream, TestPartialUpdateDeduplication, TestDynamicFieldCollectionResource, SignatureUpdateProcessorFactoryTest, TestMergePolicyConfig, EchoParamsTest, URLClassifyProcessorTest, DebugComponentTest, TestFoldingMultitermQuery, OutputWriterTest, SpellCheckCollatorTest, TestNumberUtils, FileUtilsTest, TestFieldResource, StandardRequestHandlerTest, ZkNodePropsTest, TestSolrJ, TestSchemaNameResource, NoCacheHeaderTest, TestHashPartitioner, TestBinaryField, TestSuggestSpellingConverter, SimplePostToolTest, BasicDistributedZk2Test, TestUpdate, DocValuesMultiTest, BasicZkTest, ChaosMonkeySafeLeaderTest, TestLuceneMatchVersion, CurrencyFieldXmlFileTest, OpenExchangeRatesOrgProviderTest, TestFiltering, TestManagedSchemaFieldResource, CopyFieldTest, TestArbitraryIndexDir, PathHierarchyTokenizerFactoryTest, TestDistributedGrouping, TestCSVLoader, TestAddFieldRealTimeGet, CurrencyFieldOpenExchangeTest, TestSolrQueryParserDefaultOperatorResource, IndexReaderFactoryTest, LeaderElectionIntegrationTest, TestQueryTypes, TestDefaultSearchFieldResource, TestElisionMultitermQuery, DirectSolrSpellCheckerTest, TermVectorComponentDistributedTest, CollectionsAPIDistributedZkTest, XmlUpdateRequestHandlerTest, TestLazyCores, TestDFRSimilarityFactory, SolrPluginUtilsTest, TestSchemaVersionResource, HighlighterConfigTest, ZkSolrClientTest, TestMultiCoreConfBootstrap, TestDynamicFieldResource, TestRTGBase, TestDefaultSimilarityFactory, SolrCmdDistributorTest, TestCoreDiscovery, TestComponentsName, SpellCheckComponentTest, SyncSliceTest]
[junit4:junit4] Completed on J1 in 102.95s, 1 test, 1 failure <<< FAILURES!
[...truncated 478 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1252 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 40 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
6205 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6205/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
2 tests failed.
REGRESSION: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:60797/pm_ryp/zj/onenodecollectioncore returned non ok status:404, message:Can not find: /pm_ryp/zj/onenodecollectioncore/update
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:60797/pm_ryp/zj/onenodecollectioncore returned non ok status:404, message:Can not find: /pm_ryp/zj/onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([7CD869FC4AE7998B:FD3EE7E43DB8F9B7]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:35161/collection1lastClient and got 271 from http://127.0.0.1:50339/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:35161/collection1lastClient and got 271 from http://127.0.0.1:50339/collection1
at __randomizedtesting.SeedInfo.seed([7CD869FC4AE7998B:FD3EE7E43DB8F9B7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
Build Log:
[...truncated 9426 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 600316 T2337 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /pm_ryp/zj
[junit4:junit4] 2> 600320 T2337 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355
[junit4:junit4] 2> 600321 T2337 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 600322 T2338 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 600422 T2337 oasc.ZkTestServer.run start zk server on port:56455
[junit4:junit4] 2> 600423 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 600758 T2344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bbb5c763 name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 600759 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 600760 T2337 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 600764 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 600765 T2346 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@443d4dc1 name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 600765 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 600766 T2337 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 600769 T2337 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 600771 T2337 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 600774 T2337 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 600776 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 600777 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 600781 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 600782 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 600785 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 600785 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 600788 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 600789 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 600791 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 600792 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 600795 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 600795 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 600798 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 600799 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 600802 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 600802 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 600805 T2337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 600806 T2337 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 600881 T2337 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 600883 T2337 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54340
[junit4:junit4] 2> 600884 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 600885 T2337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 600885 T2337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843
[junit4:junit4] 2> 600886 T2337 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/solr.xml
[junit4:junit4] 2> 600887 T2337 oasc.CoreContainer.<init> New CoreContainer -906283424
[junit4:junit4] 2> 600887 T2337 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/'
[junit4:junit4] 2> 600888 T2337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/'
[junit4:junit4] 2> 600925 T2337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 600925 T2337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 600926 T2337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 600927 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 600927 T2337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 600928 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 600929 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 600929 T2337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 600930 T2337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 600931 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 600935 T2337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 600936 T2337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56455/solr
[junit4:junit4] 2> 600937 T2337 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 600938 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 600939 T2357 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34728ba8 name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 600939 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 600941 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 600947 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 600948 T2359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6959a5f6 name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 600948 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 600950 T2337 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 600952 T2337 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 600954 T2337 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 600956 T2337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54340_pm_ryp%2Fzj
[junit4:junit4] 2> 600957 T2337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54340_pm_ryp%2Fzj
[junit4:junit4] 2> 600959 T2337 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 600964 T2337 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 600966 T2337 oasc.Overseer.start Overseer (id=89890861439844355-127.0.0.1:54340_pm_ryp%2Fzj-n_0000000000) starting
[junit4:junit4] 2> 600968 T2337 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 600971 T2361 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 600971 T2337 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 600974 T2337 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 600975 T2337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 600978 T2360 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 600980 T2362 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 600981 T2362 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 602481 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 602482 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54340/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:54340_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 602482 T2360 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 602482 T2360 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 602485 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 602982 T2362 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/collection1
[junit4:junit4] 2> 602983 T2362 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 602984 T2362 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 602984 T2362 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 602985 T2362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/collection1/'
[junit4:junit4] 2> 602986 T2362 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/collection1/lib/README' to classloader
[junit4:junit4] 2> 602987 T2362 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 603014 T2362 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 603042 T2362 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 603044 T2362 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 603048 T2362 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 603360 T2362 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 603365 T2362 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 603368 T2362 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 603379 T2362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 603383 T2362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 603386 T2362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 603388 T2362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 603388 T2362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 603388 T2362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 603389 T2362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 603390 T2362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 603390 T2362 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 603391 T2362 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371625693843/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/
[junit4:junit4] 2> 603391 T2362 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f2a763f
[junit4:junit4] 2> 603391 T2362 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 603392 T2362 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data
[junit4:junit4] 2> 603393 T2362 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/index/
[junit4:junit4] 2> 603393 T2362 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 603393 T2362 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/index
[junit4:junit4] 2> 603395 T2362 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60e59f68 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bc347c6),segFN=segments_1,generation=1}
[junit4:junit4] 2> 603395 T2362 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 603397 T2362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 603397 T2362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 603397 T2362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 603398 T2362 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 603398 T2362 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 603399 T2362 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 603399 T2362 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 603400 T2362 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 603400 T2362 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 603401 T2362 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 603403 T2362 oass.SolrIndexSearcher.<init> Opening Searcher@37c40f2b main
[junit4:junit4] 2> 603404 T2362 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/tlog
[junit4:junit4] 2> 603404 T2362 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 603405 T2362 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 603408 T2363 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37c40f2b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 603408 T2362 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 603408 T2362 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54340/pm_ryp/zj collection:control_collection shard:shard1
[junit4:junit4] 2> 603409 T2362 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 603416 T2362 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 603418 T2362 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 603418 T2362 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 603419 T2362 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54340/pm_ryp/zj/collection1/
[junit4:junit4] 2> 603419 T2362 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 603420 T2362 oasc.SyncStrategy.syncToMe http://127.0.0.1:54340/pm_ryp/zj/collection1/ has no replicas
[junit4:junit4] 2> 603420 T2362 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54340/pm_ryp/zj/collection1/
[junit4:junit4] 2> 603420 T2362 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 603989 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 603995 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 604027 T2362 oasc.ZkController.register We are http://127.0.0.1:54340/pm_ryp/zj/collection1/ and leader is http://127.0.0.1:54340/pm_ryp/zj/collection1/
[junit4:junit4] 2> 604027 T2362 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54340/pm_ryp/zj
[junit4:junit4] 2> 604028 T2362 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 604028 T2362 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 604028 T2362 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 604030 T2362 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 604031 T2337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 604032 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 604032 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 604036 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 604038 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 604039 T2366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7379f8d1 name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 604040 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 604041 T2337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 604043 T2337 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 604113 T2337 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 604115 T2337 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44487
[junit4:junit4] 2> 604116 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 604117 T2337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 604117 T2337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078
[junit4:junit4] 2> 604118 T2337 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/solr.xml
[junit4:junit4] 2> 604119 T2337 oasc.CoreContainer.<init> New CoreContainer -385220402
[junit4:junit4] 2> 604119 T2337 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/'
[junit4:junit4] 2> 604120 T2337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/'
[junit4:junit4] 2> 604157 T2337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 604158 T2337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 604159 T2337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 604159 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 604160 T2337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 604160 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 604161 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 604162 T2337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 604162 T2337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 604163 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 604168 T2337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 604169 T2337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56455/solr
[junit4:junit4] 2> 604169 T2337 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 604170 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 604172 T2377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a8ae3c3 name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 604172 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 604174 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 604181 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 604182 T2379 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38b751ff name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 604183 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 604186 T2337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 605190 T2337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44487_pm_ryp%2Fzj
[junit4:junit4] 2> 605192 T2337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44487_pm_ryp%2Fzj
[junit4:junit4] 2> 605195 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 605195 T2366 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 605195 T2379 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 605197 T2359 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 605201 T2380 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 605201 T2380 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 605499 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 605500 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54340/pm_ryp/zj",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:54340_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 605503 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 605504 T2360 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 605504 T2360 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 605508 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 605508 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 605508 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 606202 T2380 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/collection1
[junit4:junit4] 2> 606203 T2380 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 606204 T2380 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 606205 T2380 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 606206 T2380 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/collection1/'
[junit4:junit4] 2> 606207 T2380 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/collection1/lib/README' to classloader
[junit4:junit4] 2> 606208 T2380 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 606240 T2380 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 606269 T2380 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 606270 T2380 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 606275 T2380 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 606587 T2380 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 606592 T2380 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 606594 T2380 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 606604 T2380 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 606608 T2380 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 606612 T2380 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 606613 T2380 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 606613 T2380 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 606614 T2380 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 606615 T2380 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 606615 T2380 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 606615 T2380 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 606616 T2380 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/
[junit4:junit4] 2> 606616 T2380 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f2a763f
[junit4:junit4] 2> 606617 T2380 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 606617 T2380 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1
[junit4:junit4] 2> 606618 T2380 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/index/
[junit4:junit4] 2> 606618 T2380 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 606619 T2380 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/index
[junit4:junit4] 2> 606620 T2380 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cb254461 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9145532b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 606620 T2380 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 606622 T2380 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 606622 T2380 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 606623 T2380 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 606623 T2380 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 606624 T2380 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 606624 T2380 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 606624 T2380 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 606625 T2380 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 606625 T2380 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 606627 T2380 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 606629 T2380 oass.SolrIndexSearcher.<init> Opening Searcher@2cbbffdb main
[junit4:junit4] 2> 606629 T2380 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/tlog
[junit4:junit4] 2> 606630 T2380 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 606630 T2380 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 606634 T2381 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2cbbffdb main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 606635 T2380 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 606635 T2380 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44487/pm_ryp/zj collection:collection1 shard:shard1
[junit4:junit4] 2> 606636 T2380 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 606641 T2380 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 606643 T2380 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 606643 T2380 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 606643 T2380 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44487/pm_ryp/zj/collection1/
[junit4:junit4] 2> 606644 T2380 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 606644 T2380 oasc.SyncStrategy.syncToMe http://127.0.0.1:44487/pm_ryp/zj/collection1/ has no replicas
[junit4:junit4] 2> 606644 T2380 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44487/pm_ryp/zj/collection1/
[junit4:junit4] 2> 606645 T2380 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 607013 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 607020 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 607020 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 607020 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 607051 T2380 oasc.ZkController.register We are http://127.0.0.1:44487/pm_ryp/zj/collection1/ and leader is http://127.0.0.1:44487/pm_ryp/zj/collection1/
[junit4:junit4] 2> 607052 T2380 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44487/pm_ryp/zj
[junit4:junit4] 2> 607052 T2380 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 607052 T2380 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 607053 T2380 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 607054 T2380 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 607055 T2337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 607056 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 607057 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 607130 T2337 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 607132 T2337 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60797
[junit4:junit4] 2> 607133 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 607133 T2337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 607134 T2337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095
[junit4:junit4] 2> 607135 T2337 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/solr.xml
[junit4:junit4] 2> 607135 T2337 oasc.CoreContainer.<init> New CoreContainer -2072213267
[junit4:junit4] 2> 607136 T2337 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/'
[junit4:junit4] 2> 607137 T2337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/'
[junit4:junit4] 2> 607174 T2337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 607175 T2337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 607176 T2337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 607176 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 607177 T2337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 607178 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 607178 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 607179 T2337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 607180 T2337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 607180 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 607185 T2337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 607185 T2337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56455/solr
[junit4:junit4] 2> 607186 T2337 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 607187 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 607189 T2393 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a8be83c name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 607189 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 607191 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 607195 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 607197 T2395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf5f194c name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 607197 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 607200 T2337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 608203 T2337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60797_pm_ryp%2Fzj
[junit4:junit4] 2> 608205 T2337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60797_pm_ryp%2Fzj
[junit4:junit4] 2> 608207 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 608207 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 608207 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 608207 T2395 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 608209 T2379 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 608209 T2366 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 608209 T2359 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 608214 T2396 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 608214 T2396 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 608525 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 608526 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 608529 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60797/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:60797_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 608529 T2360 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 608530 T2360 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 608534 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 608534 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 608534 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 608534 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 609215 T2396 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/collection1
[junit4:junit4] 2> 609216 T2396 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 609217 T2396 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 609217 T2396 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 609219 T2396 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/collection1/'
[junit4:junit4] 2> 609220 T2396 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/collection1/lib/README' to classloader
[junit4:junit4] 2> 609220 T2396 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 609243 T2396 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 609285 T2396 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 609287 T2396 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 609291 T2396 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 609591 T2396 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 609596 T2396 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 609598 T2396 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 609608 T2396 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 609612 T2396 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 609615 T2396 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 609616 T2396 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 609617 T2396 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 609617 T2396 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 609618 T2396 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 609619 T2396 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 609619 T2396 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 609619 T2396 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371625700095/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/
[junit4:junit4] 2> 609620 T2396 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f2a763f
[junit4:junit4] 2> 609620 T2396 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 609621 T2396 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2
[junit4:junit4] 2> 609621 T2396 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/index/
[junit4:junit4] 2> 609622 T2396 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 609622 T2396 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/index
[junit4:junit4] 2> 609623 T2396 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ef63c26 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8f8ce8),segFN=segments_1,generation=1}
[junit4:junit4] 2> 609624 T2396 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 609625 T2396 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 609626 T2396 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 609626 T2396 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 609627 T2396 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 609627 T2396 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 609628 T2396 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 609628 T2396 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 609629 T2396 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 609629 T2396 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 609630 T2396 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 609632 T2396 oass.SolrIndexSearcher.<init> Opening Searcher@e909799f main
[junit4:junit4] 2> 609633 T2396 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/tlog
[junit4:junit4] 2> 609634 T2396 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 609634 T2396 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 609638 T2397 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e909799f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 609639 T2396 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 609640 T2396 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60797/pm_ryp/zj collection:collection1 shard:shard2
[junit4:junit4] 2> 609641 T2396 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 609645 T2396 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 609647 T2396 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 609647 T2396 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 609647 T2396 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60797/pm_ryp/zj/collection1/
[junit4:junit4] 2> 609648 T2396 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 609648 T2396 oasc.SyncStrategy.syncToMe http://127.0.0.1:60797/pm_ryp/zj/collection1/ has no replicas
[junit4:junit4] 2> 609648 T2396 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60797/pm_ryp/zj/collection1/
[junit4:junit4] 2> 609649 T2396 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 610039 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 610054 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610054 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610054 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610055 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610104 T2396 oasc.ZkController.register We are http://127.0.0.1:60797/pm_ryp/zj/collection1/ and leader is http://127.0.0.1:60797/pm_ryp/zj/collection1/
[junit4:junit4] 2> 610104 T2396 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60797/pm_ryp/zj
[junit4:junit4] 2> 610104 T2396 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 610105 T2396 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 610105 T2396 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 610107 T2396 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 610108 T2337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 610109 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 610110 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 610192 T2337 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 610194 T2337 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44392
[junit4:junit4] 2> 610195 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 610196 T2337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 610197 T2337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148
[junit4:junit4] 2> 610197 T2337 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/solr.xml
[junit4:junit4] 2> 610198 T2337 oasc.CoreContainer.<init> New CoreContainer 1870458639
[junit4:junit4] 2> 610199 T2337 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/'
[junit4:junit4] 2> 610199 T2337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/'
[junit4:junit4] 2> 610239 T2337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 610240 T2337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 610240 T2337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 610241 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 610242 T2337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 610242 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 610243 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 610244 T2337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 610244 T2337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 610245 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 610250 T2337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 610251 T2337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56455/solr
[junit4:junit4] 2> 610252 T2337 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 610253 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 610254 T2409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78514b04 name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 610254 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 610256 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 610261 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 610262 T2411 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a6c13ea name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 610263 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 610266 T2337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 611269 T2337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44392_pm_ryp%2Fzj
[junit4:junit4] 2> 611271 T2337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44392_pm_ryp%2Fzj
[junit4:junit4] 2> 611273 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 611273 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 611273 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 611273 T2411 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 611274 T2395 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 611275 T2359 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 611276 T2366 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 611276 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 611276 T2379 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 611279 T2412 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 611279 T2412 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 611559 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 611561 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60797/pm_ryp/zj",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:60797_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 611564 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44392/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:44392_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 611565 T2360 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 611565 T2360 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 611567 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 611568 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 611568 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 611568 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 611567 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 612280 T2412 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/collection1
[junit4:junit4] 2> 612281 T2412 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 612281 T2412 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 612282 T2412 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 612283 T2412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/collection1/'
[junit4:junit4] 2> 612284 T2412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/collection1/lib/README' to classloader
[junit4:junit4] 2> 612284 T2412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 612307 T2412 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 612337 T2412 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 612338 T2412 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 612342 T2412 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 612658 T2412 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 612663 T2412 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 612665 T2412 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 612675 T2412 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 612679 T2412 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 612682 T2412 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 612683 T2412 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 612684 T2412 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 612684 T2412 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 612685 T2412 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 612685 T2412 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 612686 T2412 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 612686 T2412 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371625703148/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/
[junit4:junit4] 2> 612687 T2412 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f2a763f
[junit4:junit4] 2> 612687 T2412 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 612688 T2412 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3
[junit4:junit4] 2> 612688 T2412 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index/
[junit4:junit4] 2> 612688 T2412 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 612689 T2412 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index
[junit4:junit4] 2> 612690 T2412 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@d6076caa lockFactory=org.apache.lucene.store.NativeFSLockFactory@400e96e4),segFN=segments_1,generation=1}
[junit4:junit4] 2> 612691 T2412 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 612692 T2412 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 612692 T2412 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 612693 T2412 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 612694 T2412 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 612694 T2412 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 612694 T2412 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 612695 T2412 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 612695 T2412 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 612696 T2412 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 612697 T2412 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 612699 T2412 oass.SolrIndexSearcher.<init> Opening Searcher@78c65ce0 main
[junit4:junit4] 2> 612699 T2412 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/tlog
[junit4:junit4] 2> 612700 T2412 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 612701 T2412 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 612705 T2413 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78c65ce0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 612707 T2412 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 612708 T2412 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44392/pm_ryp/zj collection:collection1 shard:shard1
[junit4:junit4] 2> 612712 T2412 oasc.ZkController.register We are http://127.0.0.1:44392/pm_ryp/zj/collection1/ and leader is http://127.0.0.1:44487/pm_ryp/zj/collection1/
[junit4:junit4] 2> 612712 T2412 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44392/pm_ryp/zj
[junit4:junit4] 2> 612713 T2412 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 612713 T2412 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1556 name=collection1 org.apache.solr.core.SolrCore@a72a270d url=http://127.0.0.1:44392/pm_ryp/zj/collection1 node=127.0.0.1:44392_pm_ryp%2Fzj C1556_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:44392/pm_ryp/zj, state=down, node_name=127.0.0.1:44392_pm_ryp%2Fzj, collection=collection1, core=collection1}
[junit4:junit4] 2> 612714 T2414 C1556 P44392 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 612714 T2412 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 612714 T2414 C1556 P44392 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 612715 T2414 C1556 P44392 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 612715 T2414 C1556 P44392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 612716 T2337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 612717 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 612716 T2414 C1556 P44392 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 612717 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 612722 T2375 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 612790 T2337 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 612792 T2337 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50858
[junit4:junit4] 2> 612793 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 612794 T2337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 612794 T2337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756
[junit4:junit4] 2> 612795 T2337 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/solr.xml
[junit4:junit4] 2> 612795 T2337 oasc.CoreContainer.<init> New CoreContainer 1628618729
[junit4:junit4] 2> 612796 T2337 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/'
[junit4:junit4] 2> 612797 T2337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/'
[junit4:junit4] 2> 612839 T2337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 612840 T2337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 612841 T2337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 612842 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 612843 T2337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 612844 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 612844 T2337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 612845 T2337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 612846 T2337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 612846 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 612851 T2337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 612852 T2337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56455/solr
[junit4:junit4] 2> 612853 T2337 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 612854 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612855 T2426 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a938e6b7 name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612856 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612857 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 612862 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612863 T2428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7435134 name:ZooKeeperConnection Watcher:127.0.0.1:56455/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612864 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612867 T2337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 613073 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 613073 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44392/pm_ryp/zj",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:44392_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 613077 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613077 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613077 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613077 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613077 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613077 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613723 T2375 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 613724 T2375 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=4&wt=javabin&nodeName=127.0.0.1:44392_pm_ryp%252Fzj&core=collection1} status=0 QTime=1002
[junit4:junit4] 2> 613870 T2337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50858_pm_ryp%2Fzj
[junit4:junit4] 2> 613872 T2337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50858_pm_ryp%2Fzj
[junit4:junit4] 2> 613874 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613874 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613874 T2395 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 613874 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613874 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613876 T2411 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 613875 T2359 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 613876 T2428 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 613875 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 613877 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 613877 T2379 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 613877 T2366 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 613881 T2429 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 613881 T2429 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 614581 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 614582 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50858/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "node_name":"127.0.0.1:50858_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 614582 T2360 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 614582 T2360 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 614586 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 614586 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 614586 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 614586 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 614586 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 614586 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 614883 T2429 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/collection1
[junit4:junit4] 2> 614883 T2429 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 614884 T2429 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 614884 T2429 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 614886 T2429 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/collection1/'
[junit4:junit4] 2> 614887 T2429 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/collection1/lib/README' to classloader
[junit4:junit4] 2> 614887 T2429 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 614910 T2429 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 614939 T2429 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 614941 T2429 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 614945 T2429 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 615260 T2429 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 615265 T2429 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 615268 T2429 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 615278 T2429 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 615282 T2429 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 615285 T2429 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 615286 T2429 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 615287 T2429 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 615287 T2429 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 615288 T2429 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 615289 T2429 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 615289 T2429 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 615289 T2429 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371625705756/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/
[junit4:junit4] 2> 615290 T2429 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f2a763f
[junit4:junit4] 2> 615290 T2429 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 615291 T2429 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4
[junit4:junit4] 2> 615291 T2429 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index/
[junit4:junit4] 2> 615292 T2429 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 615292 T2429 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index
[junit4:junit4] 2> 615294 T2429 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6d013ef0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@a5f3efaa),segFN=segments_1,generation=1}
[junit4:junit4] 2> 615294 T2429 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 615295 T2429 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 615296 T2429 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 615296 T2429 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 615297 T2429 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 615298 T2429 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 615298 T2429 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 615298 T2429 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 615299 T2429 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 615299 T2429 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 615301 T2429 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 615302 T2429 oass.SolrIndexSearcher.<init> Opening Searcher@890e8449 main
[junit4:junit4] 2> 615303 T2429 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/tlog
[junit4:junit4] 2> 615304 T2429 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 615304 T2429 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 615308 T2430 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@890e8449 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 615309 T2429 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 615310 T2429 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50858/pm_ryp/zj collection:collection1 shard:shard2
[junit4:junit4] 2> 615312 T2429 oasc.ZkController.register We are http://127.0.0.1:50858/pm_ryp/zj/collection1/ and leader is http://127.0.0.1:60797/pm_ryp/zj/collection1/
[junit4:junit4] 2> 615313 T2429 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50858/pm_ryp/zj
[junit4:junit4] 2> 615313 T2429 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 615313 T2429 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1557 name=collection1 org.apache.solr.core.SolrCore@bca8eb18 url=http://127.0.0.1:50858/pm_ryp/zj/collection1 node=127.0.0.1:50858_pm_ryp%2Fzj C1557_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:50858/pm_ryp/zj, state=down, node_name=127.0.0.1:50858_pm_ryp%2Fzj, collection=collection1, core=collection1}
[junit4:junit4] 2> 615314 T2431 C1557 P50858 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 615314 T2429 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 615314 T2431 C1557 P50858 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 615315 T2431 C1557 P50858 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 615315 T2431 C1557 P50858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 615315 T2337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 615316 T2337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 615316 T2431 C1557 P50858 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 615317 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 615322 T2390 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 615322 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 615324 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 615332 T2375 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> ASYNC NEW_CORE C1558 name=collection1 org.apache.solr.core.SolrCore@a72a270d url=http://127.0.0.1:44392/pm_ryp/zj/collection1 node=127.0.0.1:44392_pm_ryp%2Fzj C1558_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:44392/pm_ryp/zj, state=recovering, node_name=127.0.0.1:44392_pm_ryp%2Fzj, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 615725 T2414 C1558 P44392 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:44487/pm_ryp/zj/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 615726 T2414 C1558 P44392 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44392/pm_ryp/zj START replicas=[http://127.0.0.1:44487/pm_ryp/zj/collection1/] nUpdates=100
[junit4:junit4] 2> 615726 T2414 C1558 P44392 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 615726 T2414 C1558 P44392 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 615727 T2414 C1558 P44392 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 615727 T2414 C1558 P44392 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 615727 T2414 C1558 P44392 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 615728 T2414 C1558 P44392 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:44487/pm_ryp/zj/collection1/. core=collection1
[junit4:junit4] 2> 615728 T2414 C1558 P44392 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1559 name=collection1 org.apache.solr.core.SolrCore@b7d1547a url=http://127.0.0.1:44487/pm_ryp/zj/collection1 node=127.0.0.1:44487_pm_ryp%2Fzj C1559_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:44487/pm_ryp/zj, state=active, node_name=127.0.0.1:44487_pm_ryp%2Fzj, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 615732 T2372 C1559 P44487 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 615735 T2373 C1559 P44487 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 615736 T2373 C1559 P44487 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cb254461 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9145532b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 615737 T2373 C1559 P44487 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 615738 T2373 C1559 P44487 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cb254461 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9145532b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cb254461 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9145532b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 615739 T2373 C1559 P44487 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 615740 T2373 C1559 P44487 oass.SolrIndexSearcher.<init> Opening Searcher@75499f94 realtime
[junit4:junit4] 2> 615740 T2373 C1559 P44487 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 615741 T2373 C1559 P44487 oasup.LogUpdateProcessor.finish [collection1] webapp=/pm_ryp/zj path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 6
[junit4:junit4] 2> 615742 T2414 C1558 P44392 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 615742 T2414 C1558 P44392 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 615744 T2374 C1559 P44487 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 615745 T2374 C1559 P44487 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 615745 T2414 C1558 P44392 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 615746 T2414 C1558 P44392 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 615746 T2414 C1558 P44392 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 615748 T2373 C1559 P44487 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 615748 T2414 C1558 P44392 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 615749 T2414 C1558 P44392 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index.20130619070828783
[junit4:junit4] 2> 615749 T2414 C1558 P44392 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@43303cc1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@eabc19e5) fullCopy=false
[junit4:junit4] 2> 615751 T2372 C1559 P44487 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 615752 T2414 C1558 P44392 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 615753 T2414 C1558 P44392 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 615753 T2414 C1558 P44392 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 615754 T2414 C1558 P44392 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@d6076caa lockFactory=org.apache.lucene.store.NativeFSLockFactory@400e96e4),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@d6076caa lockFactory=org.apache.lucene.store.NativeFSLockFactory@400e96e4),segFN=segments_2,generation=2}
[junit4:junit4] 2> 615754 T2414 C1558 P44392 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 615755 T2414 C1558 P44392 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 615755 T2414 C1558 P44392 oass.SolrIndexSearcher.<init> Opening Searcher@ebe000a main
[junit4:junit4] 2> 615756 T2413 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ebe000a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 615757 T2414 C1558 P44392 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index.20130619070828783 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index.20130619070828783;done=true>>]
[junit4:junit4] 2> 615757 T2414 C1558 P44392 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index.20130619070828783
[junit4:junit4] 2> 615757 T2414 C1558 P44392 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index.20130619070828783
[junit4:junit4] 2> 615758 T2414 C1558 P44392 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 615758 T2414 C1558 P44392 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 615758 T2414 C1558 P44392 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 615759 T2414 C1558 P44392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 615760 T2414 C1558 P44392 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 616091 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 616092 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50858/pm_ryp/zj",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "node_name":"127.0.0.1:50858_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 616095 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 616096 T2360 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 616096 T2360 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 616099 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44392/pm_ryp/zj",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:44392_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 616102 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 616102 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 616102 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 616102 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 616102 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 616102 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 616323 T2390 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 616324 T2390 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=5&wt=javabin&nodeName=127.0.0.1:50858_pm_ryp%252Fzj&core=collection1} status=0 QTime=1002
[junit4:junit4] 2> 616334 T2375 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/onenodecollectioncore
[junit4:junit4] 2> 616335 T2375 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 616337 T2375 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 616338 T2375 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 616340 T2375 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 616341 T2375 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 616344 T2375 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 616345 T2375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/onenodecollectioncore/'
[junit4:junit4] 2> 616382 T2375 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 616422 T2375 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 616424 T2375 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 616428 T2375 oass.IndexSchema.readSchema [onenodecollectioncore] Schema name=test
[junit4:junit4] 2> 616767 T2375 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 616772 T2375 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 616775 T2375 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 616785 T2375 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 616789 T2375 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 616793 T2375 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 616795 T2375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 616796 T2375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 616796 T2375 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 616797 T2375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 616798 T2375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 616799 T2375 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 616799 T2375 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371625697078/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/
[junit4:junit4] 2> 616800 T2375 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f2a763f
[junit4:junit4] 2> 616801 T2375 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 616802 T2375 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection
[junit4:junit4] 2> 616803 T2375 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/index/
[junit4:junit4] 2> 616803 T2375 oasc.SolrCore.initIndex WARN [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 616804 T2375 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/index
[junit4:junit4] 2> 616806 T2375 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3b57ab56 lockFactory=org.apache.lucene.store.NativeFSLockFactory@58ab596c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 616807 T2375 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 616809 T2375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 616810 T2375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 616810 T2375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 616811 T2375 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 616812 T2375 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 616813 T2375 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 616813 T2375 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 616814 T2375 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 616815 T2375 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 616817 T2375 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 616819 T2375 oass.SolrIndexSearcher.<init> Opening Searcher@2e9348d6 main
[junit4:junit4] 2> 616820 T2375 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/tlog
[junit4:junit4] 2> 616821 T2375 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 616821 T2375 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 616825 T2435 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@2e9348d6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 616826 T2375 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 616826 T2375 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:44487/pm_ryp/zj collection:onenodecollection shard:shard1
[junit4:junit4] 2> 616828 T2375 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 616833 T2375 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 616835 T2375 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 616836 T2375 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 616837 T2375 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44487/pm_ryp/zj/onenodecollectioncore/
[junit4:junit4] 2> 616837 T2375 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 616838 T2375 oasc.SyncStrategy.syncToMe http://127.0.0.1:44487/pm_ryp/zj/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 616839 T2375 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44487/pm_ryp/zj/onenodecollectioncore/
[junit4:junit4] 2> 616839 T2375 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 617608 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 617615 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617615 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617616 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617615 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617616 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617615 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617646 T2375 oasc.ZkController.register We are http://127.0.0.1:44487/pm_ryp/zj/onenodecollectioncore/ and leader is http://127.0.0.1:44487/pm_ryp/zj/onenodecollectioncore/
[junit4:junit4] 2> 617647 T2375 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:44487/pm_ryp/zj
[junit4:junit4] 2> 617648 T2375 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 617648 T2375 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 617650 T2375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 617652 T2375 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection&action=CREATE&version=2&numShards=1&name=onenodecollectioncore&wt=javabin&roles=none&collection=onenodecollection} status=0 QTime=2320
[junit4:junit4] 2> 617652 T2337 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 617653 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1560 name=collection1 org.apache.solr.core.SolrCore@bca8eb18 url=http://127.0.0.1:50858/pm_ryp/zj/collection1 node=127.0.0.1:50858_pm_ryp%2Fzj C1560_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:50858/pm_ryp/zj, state=recovering, node_name=127.0.0.1:50858_pm_ryp%2Fzj, collection=collection1, shard=shard2, core=collection1}
[junit4:junit4] 2> 618325 T2431 C1560 P50858 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:60797/pm_ryp/zj/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 618325 T2431 C1560 P50858 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50858/pm_ryp/zj START replicas=[http://127.0.0.1:60797/pm_ryp/zj/collection1/] nUpdates=100
[junit4:junit4] 2> 618326 T2431 C1560 P50858 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 618326 T2431 C1560 P50858 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 618326 T2431 C1560 P50858 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 618326 T2431 C1560 P50858 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 618327 T2431 C1560 P50858 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 618327 T2431 C1560 P50858 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:60797/pm_ryp/zj/collection1/. core=collection1
[junit4:junit4] 2> 618327 T2431 C1560 P50858 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1561 name=collection1 org.apache.solr.core.SolrCore@cfb8d063 url=http://127.0.0.1:60797/pm_ryp/zj/collection1 node=127.0.0.1:60797_pm_ryp%2Fzj C1561_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:60797/pm_ryp/zj, state=active, node_name=127.0.0.1:60797_pm_ryp%2Fzj, collection=collection1, shard=shard2, core=collection1, leader=true}
[junit4:junit4] 2> 618332 T2388 C1561 P60797 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 618332 T2391 C1561 P60797 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 618335 T2391 C1561 P60797 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ef63c26 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8f8ce8),segFN=segments_1,generation=1}
[junit4:junit4] 2> 618337 T2391 C1561 P60797 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 618338 T2391 C1561 P60797 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ef63c26 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8f8ce8),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3ef63c26 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8f8ce8),segFN=segments_2,generation=2}
[junit4:junit4] 2> 618339 T2391 C1561 P60797 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 618341 T2391 C1561 P60797 oass.SolrIndexSearcher.<init> Opening Searcher@2c33ce55 realtime
[junit4:junit4] 2> 618342 T2391 C1561 P60797 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 618343 T2391 C1561 P60797 oasup.LogUpdateProcessor.finish [collection1] webapp=/pm_ryp/zj path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 11
[junit4:junit4] 2> 618344 T2431 C1560 P50858 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 618345 T2431 C1560 P50858 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 618346 T2389 C1561 P60797 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 618347 T2389 C1561 P60797 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 618348 T2431 C1560 P50858 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 618348 T2431 C1560 P50858 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 618348 T2431 C1560 P50858 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 618350 T2388 C1561 P60797 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 618351 T2431 C1560 P50858 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 618351 T2431 C1560 P50858 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index.20130619070831385
[junit4:junit4] 2> 618352 T2431 C1560 P50858 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@87a51688 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9d97805d) fullCopy=false
[junit4:junit4] 2> 618354 T2391 C1561 P60797 oasc.SolrCore.execute [collection1] webapp=/pm_ryp/zj path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 618355 T2431 C1560 P50858 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 618356 T2431 C1560 P50858 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 618356 T2431 C1560 P50858 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 618357 T2431 C1560 P50858 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6d013ef0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@a5f3efaa),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6d013ef0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@a5f3efaa),segFN=segments_2,generation=2}
[junit4:junit4] 2> 618357 T2431 C1560 P50858 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 618358 T2431 C1560 P50858 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 618358 T2431 C1560 P50858 oass.SolrIndexSearcher.<init> Opening Searcher@a14b6383 main
[junit4:junit4] 2> 618359 T2430 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a14b6383 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 618359 T2431 C1560 P50858 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index.20130619070831385 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index.20130619070831385;done=true>>]
[junit4:junit4] 2> 618360 T2431 C1560 P50858 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index.20130619070831385
[junit4:junit4] 2> 618360 T2431 C1560 P50858 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty4/index.20130619070831385
[junit4:junit4] 2> 618360 T2431 C1560 P50858 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 618360 T2431 C1560 P50858 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 618361 T2431 C1560 P50858 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 618361 T2431 C1560 P50858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 618362 T2431 C1560 P50858 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 618655 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 619122 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 619123 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44487_pm_ryp%2Fzj_onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 619127 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50858/pm_ryp/zj",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "node_name":"127.0.0.1:50858_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 619130 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619131 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619130 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619131 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619130 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619131 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619657 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 619658 T2337 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 619659 T2337 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 619665 T2337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 619666 T2439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ca41fe1 name:ZooKeeperConnection Watcher:127.0.0.1:56455 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 619667 T2337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89890861439844355-127.0.0.1:54340_pm_ryp%2Fzj-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (5)
[junit4:junit4] 1> /solr/overseer_elect/election/89890861439844362-127.0.0.1:44392_pm_ryp%2Fzj-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890861439844358-127.0.0.1:44487_pm_ryp%2Fzj-n_0000000001 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890861439844360-127.0.0.1:60797_pm_ryp%2Fzj-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890861439844364-127.0.0.1:50858_pm_ryp%2Fzj-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890861439844355-127.0.0.1:54340_pm_ryp%2Fzj-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89890861439844355-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:54340/pm_ryp/zj",
[junit4:junit4] 1> "node_name":"127.0.0.1:54340_pm_ryp%2Fzj",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890861439844362-4-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890861439844358-2-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89890861439844364-5-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89890861439844360-3-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 1> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60797/pm_ryp/zj",
[junit4:junit4] 1> "node_name":"127.0.0.1:60797_pm_ryp%2Fzj",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89890861439844358-127.0.0.1:44487_pm_ryp%2Fzj_onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 1> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (5)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:60797_pm_ryp%2Fzj (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:44487_pm_ryp%2Fzj (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:44392_pm_ryp%2Fzj (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50858_pm_ryp%2Fzj (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:54340_pm_ryp%2Fzj (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:54340/pm_ryp/zj",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:54340_pm_ryp%2Fzj",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "onenodecollection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:44487_pm_ryp%2Fzj_onenodecollectioncore":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 1> "roles":"none",
[junit4:junit4] 1> "collection":"onenodecollection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"onenodecollectioncore",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{
[junit4:junit4] 1> "shard1":{
[junit4:junit4] 1> "range":"80000000-ffffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "2":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "4":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44392/pm_ryp/zj",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:44392_pm_ryp%2Fzj",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}},
[junit4:junit4] 1> "shard2":{
[junit4:junit4] 1> "range":"0-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "3":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60797/pm_ryp/zj",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:60797_pm_ryp%2Fzj",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "5":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50858/pm_ryp/zj",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:50858_pm_ryp%2Fzj",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 2> 619712 T2337 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 54340
[junit4:junit4] 2> 619712 T2337 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-906283424
[junit4:junit4] 2> 620635 T2360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 620636 T2360 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54340/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:54340_pm_ryp%2Fzj",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 620639 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 620639 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 620639 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 620639 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 620639 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 621714 T2337 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 621715 T2337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 621718 T2337 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4588993b
[junit4:junit4] 2> 621722 T2337 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 621724 T2337 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 621724 T2337 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 621725 T2337 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 621726 T2337 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 621727 T2337 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 621728 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/index;done=false>>]
[junit4:junit4] 2> 621728 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data/index
[junit4:junit4] 2> 621729 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data;done=false>>]
[junit4:junit4] 2> 621729 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/control/data
[junit4:junit4] 2> 621730 T2360 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89890861439844355-127.0.0.1:54340_pm_ryp%2Fzj-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 621731 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 621731 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 621731 T2395 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 621732 T2428 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 621731 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 621732 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 621732 T2359 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 621733 T2379 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 621734 T2359 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 621734 T2411 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 621733 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 621735 T2379 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 621736 T2379 oasc.Overseer.start Overseer (id=89890861439844358-127.0.0.1:44487_pm_ryp%2Fzj-n_0000000001) starting
[junit4:junit4] 2> 621739 T2441 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 621740 T2440 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 621741 T2440 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 621742 T2440 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54340/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:54340_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 621744 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 621744 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 621744 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 621745 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 621752 T2337 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/pm_ryp/zj,null}
[junit4:junit4] 2> 621805 T2337 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 44487
[junit4:junit4] 2> 621805 T2337 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-385220402
[junit4:junit4] 2> 623249 T2440 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 623250 T2440 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44487_pm_ryp%2Fzj_onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 623253 T2440 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 623256 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 623256 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 623256 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 623256 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 624808 T2337 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 624808 T2337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 624810 T2337 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 624811 T2337 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@b7d1547a
[junit4:junit4] 2> 624814 T2337 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 624815 T2337 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 624815 T2337 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 624816 T2337 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 624817 T2337 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 624818 T2337 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 624818 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1;done=false>>]
[junit4:junit4] 2> 624819 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1
[junit4:junit4] 2> 624819 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/index;done=false>>]
[junit4:junit4] 2> 624820 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty1/index
[junit4:junit4] 2> 624821 T2337 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@88d5b7f8
[junit4:junit4] 2> 624825 T2337 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 624826 T2337 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 624826 T2337 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 624827 T2337 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 624828 T2337 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 624828 T2337 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 624829 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection;done=false>>]
[junit4:junit4] 2> 624829 T2337 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection
[junit4:junit4] 2> 624830 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/index;done=false>>]
[junit4:junit4] 2> 624830 T2337 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1371625693355/onenodecollection/index
[junit4:junit4] 2> 624831 T2440 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89890861439844358-127.0.0.1:44487_pm_ryp%2Fzj-n_0000000001) am no longer a leader.
[junit4:junit4] 2> 624832 T2379 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 624833 T2428 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 624833 T2395 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 624833 T2379 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 624833 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 624834 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 624834 T2379 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 624834 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 624837 T2411 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 624837 T2411 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 624837 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 624837 T2395 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 624838 T2411 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44392/pm_ryp/zj/collection1/
[junit4:junit4] 2> 624838 T2411 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44392/pm_ryp/zj START replicas=[http://127.0.0.1:44487/pm_ryp/zj/collection1/] nUpdates=100
[junit4:junit4] 2> 624839 T2411 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44392/pm_ryp/zj DONE. We have no versions. sync failed.
[junit4:junit4] 2> 624839 T2411 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 624840 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 624840 T2411 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44392/pm_ryp/zj/collection1/
[junit4:junit4] 2> 624840 T2395 oasc.Overseer.start Overseer (id=89890861439844360-127.0.0.1:60797_pm_ryp%2Fzj-n_0000000002) starting
[junit4:junit4] 2> 624840 T2411 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 624844 T2444 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 624844 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 624845 T2443 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 624846 T2411 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 624846 T2443 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 624847 T2443 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 624849 T2443 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44487/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44487_pm_ryp%2Fzj_onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:44487_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 624853 T2337 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/pm_ryp/zj,null}
[junit4:junit4] 2> 624856 T2395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 624856 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 624856 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 624906 T2337 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60797
[junit4:junit4] 2> 624907 T2337 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-2072213267
[junit4:junit4] 2> 625909 T2337 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 625910 T2337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 625911 T2337 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@cfb8d063
[junit4:junit4] 2> 625914 T2337 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 625915 T2337 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 625915 T2337 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 625916 T2337 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 625917 T2337 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 625918 T2337 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 625918 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2;done=false>>]
[junit4:junit4] 2> 625919 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2
[junit4:junit4] 2> 625919 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/index;done=false>>]
[junit4:junit4] 2> 625920 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty2/index
[junit4:junit4] 2> 625921 T2443 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89890861439844360-127.0.0.1:60797_pm_ryp%2Fzj-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 625922 T2428 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 625923 T2411 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 625924 T2428 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 625925 T2411 oasc.Overseer.start Overseer (id=89890861439844362-127.0.0.1:44392_pm_ryp%2Fzj-n_0000000003) starting
[junit4:junit4] 2> 625925 T2428 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 625925 T2428 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 625926 T2428 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50858/pm_ryp/zj/collection1/
[junit4:junit4] 2> 625926 T2428 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50858/pm_ryp/zj START replicas=[http://127.0.0.1:60797/pm_ryp/zj/collection1/] nUpdates=100
[junit4:junit4] 2> 625927 T2428 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50858/pm_ryp/zj DONE. We have no versions. sync failed.
[junit4:junit4] 2> 625927 T2428 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 625928 T2428 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 625928 T2428 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50858/pm_ryp/zj/collection1/
[junit4:junit4] 2> 625928 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 625928 T2447 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 625929 T2428 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 625942 T2337 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/pm_ryp/zj,null}
[junit4:junit4] 2> 625942 T2411 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 625943 T2446 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 625945 T2428 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 625945 T2446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 625945 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 625946 T2446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60797/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:60797_pm_ryp%2Fzj",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 625948 T2446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60797/pm_ryp/zj",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:60797_pm_ryp%2Fzj",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 625954 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 625954 T2411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 625995 T2337 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 44392
[junit4:junit4] 2> 625996 T2337 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1870458639
[junit4:junit4] 2> 626998 T2337 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 626999 T2337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 627001 T2337 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@a72a270d
[junit4:junit4] 2> 627004 T2337 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 627005 T2337 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 627006 T2337 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 627006 T2337 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 627007 T2337 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 627008 T2337 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 627009 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3;done=false>>]
[junit4:junit4] 2> 627009 T2337 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3
[junit4:junit4] 2> 627010 T2337 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371625693354/jetty3/index;done=false>>]
[junit4:junit4] 2> 627010 T2337 oasc.CachingDirectoryFacto
[...truncated too long message...]
(1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89890874676609027-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39624",
[junit4:junit4] 1> "node_name":"127.0.0.1:39624_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890874676609038-2-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890874676609034-4-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890874676609036-5-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50339",
[junit4:junit4] 1> "node_name":"127.0.0.1:50339_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:51326_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:35161_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:39624_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50339_ (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39624",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:39624_",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "2":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:35161",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:35161_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"},
[junit4:junit4] 1> "3":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:32920",
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "node_name":"127.0.0.1:32920_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"},
[junit4:junit4] 1> "4":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50339",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:50339_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "5":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:51326",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:51326_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=7CD869FC4AE7998B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=kk_KZ -Dtests.timezone=Europe/Belgrade -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 102s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:35161/collection1lastClient and got 271 from http://127.0.0.1:50339/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([7CD869FC4AE7998B:FD3EE7E43DB8F9B7]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2> 904363 T2861 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 102080 T2860 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=kk_KZ, timezone=Europe/Belgrade
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=4183840,total=202506240
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestPseudoReturnFields, DistributedQueryElevationComponentTest, TestRandomDVFaceting, TestFunctionQuery, LeaderElectionTest, ChaosMonkeyNothingIsSafeTest, TestQueryUtils, SchemaVersionSpecificBehaviorTest, TestSweetSpotSimilarityFactory, DocValuesTest, FileBasedSpellCheckerTest, TestPhraseSuggestions, ResourceLoaderTest, QueryElevationComponentTest, FastVectorHighlighterTest, TestSolrQueryParser, ExternalFileFieldSortTest, StatelessScriptUpdateProcessorFactoryTest, TestConfig, SolrTestCaseJ4Test, TestQuerySenderListener, RecoveryZkTest, SOLR749Test, AutoCommitTest, OverseerCollectionProcessorTest, TestFuzzyAnalyzedSuggestions, JsonLoaderTest, TestIBSimilarityFactory, TestDocSet, DistributedTermsComponentTest, TestManagedSchemaFieldResource, FieldMutatingUpdateProcessorTest, PrimUtilsTest, TestGroupingSearch, LegacyHTMLStripCharFilterTest, TestFaceting, TestSolrDeletionPolicy1, TestXIncludeConfig, TestJoin, TestReplicationHandler, LoggingHandlerTest, SpatialFilterTest, ShowFileRequestHandlerTest, IndexSchemaTest, TestSolrCoreProperties, TestFastLRUCache, TestTrie, TestDynamicFieldResource, PeerSyncTest, TestLFUCache, BasicFunctionalityTest, QueryParsingTest, AnalysisAfterCoreReloadTest, TestSurroundQueryParser, SuggesterTest, DirectSolrConnectionTest, TestCloudManagedSchema, WordBreakSolrSpellCheckerTest, AlternateDirectoryTest, SolrCoreTest, TestSerializedLuceneMatchVersion, TestStressRecovery, TestCoreContainer, BadCopyFieldTest, DateFieldTest, TestSolrQueryParserDefaultOperatorResource, DocumentBuilderTest, TestSort, ReturnFieldsTest, MinimalSchemaTest, UUIDFieldTest, BasicDistributedZkTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 102.87s, 1 test, 1 failure <<< FAILURES!
[...truncated 549 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:379: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1252 tests, 1 error, 1 failure, 14 ignored (8 assumptions)
Total time: 42 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6204 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6204/
Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 34 from http://127.0.0.1:54590/collection1lastClient and got 32 from http://127.0.0.1:56591/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 34 from http://127.0.0.1:54590/collection1lastClient and got 32 from http://127.0.0.1:56591/collection1
at __randomizedtesting.SeedInfo.seed([DBD5C3FDF836292C:5A334DE58F694910]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:55790/_/m/collection1lastClient and got 250 from http://127.0.0.1:46392/_/m/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:55790/_/m/collection1lastClient and got 250 from http://127.0.0.1:46392/_/m/collection1
at __randomizedtesting.SeedInfo.seed([DBD5C3FDF836292C:5A334DE58F694910]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9608 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 502201 T1883 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 502208 T1883 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371620915984
[junit4:junit4] 2> 502209 T1883 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 502210 T1884 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 502310 T1883 oasc.ZkTestServer.run start zk server on port:42979
[junit4:junit4] 2> 502311 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 502314 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78ff1f36 name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 502315 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 502315 T1883 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 502318 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 502319 T1892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ecbc929 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 502320 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 502320 T1883 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 502323 T1883 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 502324 T1883 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 502326 T1883 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 502328 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 502328 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 502331 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 502332 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 502334 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 502334 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 502336 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 502336 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 502338 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 502339 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 502340 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 502341 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 502342 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 502343 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 502345 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 502345 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 502347 T1883 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 502347 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 502421 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 502424 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55201
[junit4:junit4] 2> 502425 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 502425 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 502426 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125
[junit4:junit4] 2> 502426 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/solr.xml
[junit4:junit4] 2> 502427 T1883 oasc.CoreContainer.<init> New CoreContainer 1751164984
[junit4:junit4] 2> 502427 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/'
[junit4:junit4] 2> 502427 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/'
[junit4:junit4] 2> 502470 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 502471 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 502471 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 502472 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 502472 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 502472 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 502473 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 502473 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 502473 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 502474 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 502477 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 502478 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 502478 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 502479 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 502480 T1903 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f772fb0 name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 502481 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 502483 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 502497 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 502499 T1905 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2445479 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 502499 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 502501 T1883 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 502503 T1883 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 502506 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 502508 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55201_
[junit4:junit4] 2> 502510 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55201_
[junit4:junit4] 2> 502512 T1883 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 502516 T1883 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 502518 T1883 oasc.Overseer.start Overseer (id=89890548350713859-127.0.0.1:55201_-n_0000000000) starting
[junit4:junit4] 2> 502521 T1883 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 502524 T1907 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 502524 T1883 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 502527 T1883 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 502528 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 502530 T1906 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 502533 T1908 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 502533 T1908 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 504033 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 504034 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55201_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55201"}
[junit4:junit4] 2> 504034 T1906 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 504034 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 504037 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 504534 T1908 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/collection1
[junit4:junit4] 2> 504534 T1908 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 504535 T1908 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 504535 T1908 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 504536 T1908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/collection1/'
[junit4:junit4] 2> 504537 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/collection1/lib/README' to classloader
[junit4:junit4] 2> 504537 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 504565 T1908 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 504604 T1908 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 504606 T1908 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 504613 T1908 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 504967 T1908 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 504972 T1908 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 504974 T1908 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 504985 T1908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 504989 T1908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 504992 T1908 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 504992 T1908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 504993 T1908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 504993 T1908 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 504994 T1908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 504994 T1908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 504994 T1908 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 505001 T1908 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371620916125/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/control/data/
[junit4:junit4] 2> 505001 T1908 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 505001 T1908 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 505001 T1908 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/control/data
[junit4:junit4] 2> 505002 T1908 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/control/data/index/
[junit4:junit4] 2> 505002 T1908 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 505002 T1908 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/control/data/index
[junit4:junit4] 2> 505008 T1908 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 505009 T1908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 505010 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 505011 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 505011 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 505012 T1908 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 505012 T1908 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 505012 T1908 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 505012 T1908 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 505013 T1908 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 505013 T1908 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 505015 T1908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 505018 T1908 oass.SolrIndexSearcher.<init> Opening Searcher@6602ad6c main
[junit4:junit4] 2> 505018 T1908 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 505019 T1908 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 505021 T1909 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6602ad6c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 505021 T1908 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 505022 T1908 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55201 collection:control_collection shard:shard1
[junit4:junit4] 2> 505022 T1908 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 505029 T1908 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 505031 T1908 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 505031 T1908 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 505032 T1908 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55201/collection1/
[junit4:junit4] 2> 505032 T1908 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 505032 T1908 oasc.SyncStrategy.syncToMe http://127.0.0.1:55201/collection1/ has no replicas
[junit4:junit4] 2> 505032 T1908 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55201/collection1/
[junit4:junit4] 2> 505032 T1908 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 505540 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 505548 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 505590 T1908 oasc.ZkController.register We are http://127.0.0.1:55201/collection1/ and leader is http://127.0.0.1:55201/collection1/
[junit4:junit4] 2> 505590 T1908 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55201
[junit4:junit4] 2> 505590 T1908 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 505590 T1908 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 505590 T1908 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 505592 T1908 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 505593 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 505594 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 505594 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 505597 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 505598 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 505600 T1912 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a8d368 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 505600 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 505602 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 505604 T1883 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 505666 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 505668 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50699
[junit4:junit4] 2> 505669 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 505669 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 505669 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379
[junit4:junit4] 2> 505670 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/solr.xml
[junit4:junit4] 2> 505670 T1883 oasc.CoreContainer.<init> New CoreContainer 685396563
[junit4:junit4] 2> 505670 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/'
[junit4:junit4] 2> 505671 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/'
[junit4:junit4] 2> 505720 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 505720 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 505721 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 505721 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 505721 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 505722 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 505722 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 505723 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 505723 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 505723 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 505727 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 505727 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 505728 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 505728 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 505730 T1923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d4efece name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 505731 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 505732 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 505736 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 505738 T1925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f87d68c name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 505738 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 505742 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 506745 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50699_
[junit4:junit4] 2> 506747 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50699_
[junit4:junit4] 2> 506749 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 506750 T1912 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 506750 T1925 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 506751 T1905 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 506755 T1926 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 506755 T1926 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 507052 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 507053 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:55201__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55201_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55201"}
[junit4:junit4] 2> 507057 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50699_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50699"}
[junit4:junit4] 2> 507057 T1906 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 507057 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 507060 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 507060 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 507060 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 507757 T1926 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/collection1
[junit4:junit4] 2> 507757 T1926 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 507758 T1926 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 507758 T1926 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 507759 T1926 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/collection1/'
[junit4:junit4] 2> 507760 T1926 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/collection1/lib/README' to classloader
[junit4:junit4] 2> 507760 T1926 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 507802 T1926 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 507847 T1926 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 507854 T1926 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 507863 T1926 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 508362 T1926 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 508370 T1926 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 508373 T1926 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 508387 T1926 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 508391 T1926 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 508394 T1926 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 508395 T1926 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 508396 T1926 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 508396 T1926 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 508397 T1926 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 508397 T1926 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 508398 T1926 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 508398 T1926 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371620919379/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/
[junit4:junit4] 2> 508398 T1926 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 508398 T1926 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 508399 T1926 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1
[junit4:junit4] 2> 508399 T1926 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index/
[junit4:junit4] 2> 508399 T1926 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 508400 T1926 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index
[junit4:junit4] 2> 508403 T1926 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 508403 T1926 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 508405 T1926 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 508405 T1926 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 508406 T1926 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 508406 T1926 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 508407 T1926 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 508407 T1926 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 508407 T1926 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 508407 T1926 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 508408 T1926 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 508410 T1926 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 508413 T1926 oass.SolrIndexSearcher.<init> Opening Searcher@7390e5f main
[junit4:junit4] 2> 508413 T1926 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 508413 T1926 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 508417 T1927 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7390e5f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 508418 T1926 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 508419 T1926 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50699 collection:collection1 shard:shard1
[junit4:junit4] 2> 508419 T1926 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 508424 T1926 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 508426 T1926 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 508427 T1926 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 508427 T1926 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50699/collection1/
[junit4:junit4] 2> 508427 T1926 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 508427 T1926 oasc.SyncStrategy.syncToMe http://127.0.0.1:50699/collection1/ has no replicas
[junit4:junit4] 2> 508428 T1926 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50699/collection1/
[junit4:junit4] 2> 508428 T1926 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 508565 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 508573 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 508573 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 508573 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 508583 T1926 oasc.ZkController.register We are http://127.0.0.1:50699/collection1/ and leader is http://127.0.0.1:50699/collection1/
[junit4:junit4] 2> 508583 T1926 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50699
[junit4:junit4] 2> 508583 T1926 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 508584 T1926 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 508584 T1926 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 508585 T1926 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 508586 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 508586 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 508587 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 508651 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 508653 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38379
[junit4:junit4] 2> 508654 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 508654 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 508654 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365
[junit4:junit4] 2> 508655 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/solr.xml
[junit4:junit4] 2> 508655 T1883 oasc.CoreContainer.<init> New CoreContainer 702094038
[junit4:junit4] 2> 508655 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/'
[junit4:junit4] 2> 508656 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/'
[junit4:junit4] 2> 508710 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 508711 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 508711 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 508711 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 508712 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 508712 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 508712 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 508713 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 508713 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 508713 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 508716 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 508717 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 508717 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 508719 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 508720 T1939 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dadc5ea name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 508721 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 508723 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 508726 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 508728 T1941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c8b6365 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 508728 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 508731 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 509734 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38379_
[junit4:junit4] 2> 509735 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38379_
[junit4:junit4] 2> 509736 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 509738 T1912 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 509738 T1941 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 509738 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 509738 T1925 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 509739 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 509739 T1905 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 509743 T1942 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 509744 T1942 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 510078 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 510079 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50699__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50699_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50699"}
[junit4:junit4] 2> 510082 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38379_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38379"}
[junit4:junit4] 2> 510082 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 510082 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 510108 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 510108 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 510108 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 510108 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 510745 T1942 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/collection1
[junit4:junit4] 2> 510745 T1942 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 510746 T1942 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 510746 T1942 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 510747 T1942 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/collection1/'
[junit4:junit4] 2> 510747 T1942 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/collection1/lib/README' to classloader
[junit4:junit4] 2> 510748 T1942 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 510778 T1942 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 510805 T1942 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 510806 T1942 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 510811 T1942 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 511139 T1942 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 511144 T1942 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 511146 T1942 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 511157 T1942 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 511160 T1942 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 511162 T1942 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 511163 T1942 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 511164 T1942 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 511164 T1942 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 511165 T1942 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 511165 T1942 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 511165 T1942 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 511166 T1942 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371620922365/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/
[junit4:junit4] 2> 511166 T1942 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 511166 T1942 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 511167 T1942 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2
[junit4:junit4] 2> 511167 T1942 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index/
[junit4:junit4] 2> 511167 T1942 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 511168 T1942 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index
[junit4:junit4] 2> 511170 T1942 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 511171 T1942 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 511173 T1942 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 511173 T1942 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 511174 T1942 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 511174 T1942 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 511174 T1942 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 511175 T1942 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 511175 T1942 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 511175 T1942 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 511176 T1942 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 511179 T1942 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 511181 T1942 oass.SolrIndexSearcher.<init> Opening Searcher@75b2dc4f main
[junit4:junit4] 2> 511182 T1942 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 511182 T1942 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 511188 T1942 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 511188 T1942 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38379 collection:collection1 shard:shard2
[junit4:junit4] 2> 511189 T1942 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 511191 T1943 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@75b2dc4f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 511197 T1942 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 511199 T1942 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 511199 T1942 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 511199 T1942 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38379/collection1/
[junit4:junit4] 2> 511199 T1942 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 511200 T1942 oasc.SyncStrategy.syncToMe http://127.0.0.1:38379/collection1/ has no replicas
[junit4:junit4] 2> 511200 T1942 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38379/collection1/
[junit4:junit4] 2> 511200 T1942 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 511614 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 511620 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 511620 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 511620 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 511620 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 511662 T1942 oasc.ZkController.register We are http://127.0.0.1:38379/collection1/ and leader is http://127.0.0.1:38379/collection1/
[junit4:junit4] 2> 511662 T1942 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38379
[junit4:junit4] 2> 511662 T1942 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 511663 T1942 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 511663 T1942 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 511665 T1942 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 511666 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 511666 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 511667 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 511731 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 511733 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55543
[junit4:junit4] 2> 511735 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 511735 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 511735 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445
[junit4:junit4] 2> 511736 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/solr.xml
[junit4:junit4] 2> 511736 T1883 oasc.CoreContainer.<init> New CoreContainer 1264836411
[junit4:junit4] 2> 511736 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/'
[junit4:junit4] 2> 511737 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/'
[junit4:junit4] 2> 511776 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 511777 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 511777 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 511777 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 511778 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 511778 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 511778 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 511779 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 511779 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 511779 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 511782 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 511783 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 511783 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 511784 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 511785 T1955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53d5b34b name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 511786 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 511789 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 511793 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 511795 T1957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@385ad086 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 511795 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 511799 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 512801 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55543_
[junit4:junit4] 2> 512803 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55543_
[junit4:junit4] 2> 512805 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 512807 T1912 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 512807 T1925 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 512805 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 512807 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 512807 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 512807 T1957 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 512808 T1905 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 512809 T1941 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 512820 T1958 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 512820 T1958 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 513125 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 513126 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38379__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38379_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38379"}
[junit4:junit4] 2> 513129 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55543_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55543"}
[junit4:junit4] 2> 513129 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 513129 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 513132 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 513132 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 513132 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 513132 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 513132 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 513821 T1958 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/collection1
[junit4:junit4] 2> 513821 T1958 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 513822 T1958 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 513822 T1958 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 513823 T1958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/collection1/'
[junit4:junit4] 2> 513823 T1958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/collection1/lib/README' to classloader
[junit4:junit4] 2> 513824 T1958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 513848 T1958 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 513883 T1958 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 513885 T1958 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 513890 T1958 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 514201 T1958 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 514217 T1958 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 514219 T1958 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 514231 T1958 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 514234 T1958 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 514237 T1958 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 514238 T1958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 514239 T1958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 514239 T1958 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 514240 T1958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 514240 T1958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 514241 T1958 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 514241 T1958 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371620925445/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty3/
[junit4:junit4] 2> 514241 T1958 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 514242 T1958 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 514242 T1958 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty3
[junit4:junit4] 2> 514242 T1958 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty3/index/
[junit4:junit4] 2> 514242 T1958 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 514243 T1958 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty3/index
[junit4:junit4] 2> 514245 T1958 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 514246 T1958 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 514247 T1958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 514248 T1958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 514248 T1958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 514249 T1958 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 514249 T1958 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 514249 T1958 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 514249 T1958 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 514250 T1958 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 514250 T1958 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 514252 T1958 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 514255 T1958 oass.SolrIndexSearcher.<init> Opening Searcher@1b94cd65 main
[junit4:junit4] 2> 514256 T1958 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 514256 T1958 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 514260 T1959 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b94cd65 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 514263 T1958 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 514263 T1958 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55543 collection:collection1 shard:shard3
[junit4:junit4] 2> 514264 T1958 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 514269 T1958 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 514271 T1958 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 514271 T1958 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 514271 T1958 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55543/collection1/
[junit4:junit4] 2> 514271 T1958 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 514272 T1958 oasc.SyncStrategy.syncToMe http://127.0.0.1:55543/collection1/ has no replicas
[junit4:junit4] 2> 514272 T1958 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55543/collection1/
[junit4:junit4] 2> 514272 T1958 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 514637 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 514643 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 514643 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 514644 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 514643 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 514643 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 514677 T1958 oasc.ZkController.register We are http://127.0.0.1:55543/collection1/ and leader is http://127.0.0.1:55543/collection1/
[junit4:junit4] 2> 514678 T1958 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55543
[junit4:junit4] 2> 514678 T1958 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 514678 T1958 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 514678 T1958 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 514680 T1958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 514681 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 514681 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 514682 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 514754 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 514757 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33300
[junit4:junit4] 2> 514759 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 514760 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 514760 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460
[junit4:junit4] 2> 514761 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/solr.xml
[junit4:junit4] 2> 514761 T1883 oasc.CoreContainer.<init> New CoreContainer 549843637
[junit4:junit4] 2> 514761 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/'
[junit4:junit4] 2> 514762 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/'
[junit4:junit4] 2> 514816 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 514817 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 514817 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 514818 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 514818 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 514818 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 514819 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 514819 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 514819 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 514820 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 514823 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 514824 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 514824 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 514825 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 514826 T1971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20a17c88 name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 514827 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 514829 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 514834 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 514836 T1973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20bb99a0 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 514836 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 514840 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 515843 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33300_
[junit4:junit4] 2> 515845 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33300_
[junit4:junit4] 2> 515847 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 515847 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 515848 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 515848 T1925 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 515849 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 515849 T1912 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 515849 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 515850 T1973 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 515851 T1957 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 515851 T1941 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 515851 T1905 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 515854 T1974 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 515855 T1974 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 516148 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 516149 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:55543__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55543_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55543"}
[junit4:junit4] 2> 516152 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33300_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33300"}
[junit4:junit4] 2> 516152 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 516153 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 516156 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 516157 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 516156 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 516156 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 516156 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 516156 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 516856 T1974 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/collection1
[junit4:junit4] 2> 516856 T1974 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 516857 T1974 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 516858 T1974 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 516859 T1974 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/collection1/'
[junit4:junit4] 2> 516859 T1974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/collection1/lib/README' to classloader
[junit4:junit4] 2> 516860 T1974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 516884 T1974 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 516918 T1974 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 516919 T1974 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 516925 T1974 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 517279 T1974 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 517284 T1974 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 517287 T1974 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 517299 T1974 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 517302 T1974 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 517305 T1974 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 517306 T1974 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 517306 T1974 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 517307 T1974 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 517308 T1974 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 517308 T1974 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 517308 T1974 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 517308 T1974 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371620928460/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/
[junit4:junit4] 2> 517308 T1974 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 517309 T1974 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 517309 T1974 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4
[junit4:junit4] 2> 517309 T1974 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index/
[junit4:junit4] 2> 517310 T1974 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 517310 T1974 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index
[junit4:junit4] 2> 517330 T1974 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 517331 T1974 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 517333 T1974 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 517333 T1974 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 517333 T1974 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 517334 T1974 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 517334 T1974 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 517335 T1974 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 517335 T1974 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 517335 T1974 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 517336 T1974 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 517338 T1974 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 517341 T1974 oass.SolrIndexSearcher.<init> Opening Searcher@474c490a main
[junit4:junit4] 2> 517341 T1974 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 517341 T1974 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 517345 T1975 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@474c490a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 517346 T1974 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 517346 T1974 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33300 collection:collection1 shard:shard1
[junit4:junit4] 2> 517349 T1974 oasc.ZkController.register We are http://127.0.0.1:33300/collection1/ and leader is http://127.0.0.1:50699/collection1/
[junit4:junit4] 2> 517349 T1974 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33300
[junit4:junit4] 2> 517349 T1974 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 517349 T1974 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1111 name=collection1 org.apache.solr.core.SolrCore@769b944a url=http://127.0.0.1:33300/collection1 node=127.0.0.1:33300_ C1111_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:33300_, base_url=http://127.0.0.1:33300}
[junit4:junit4] 2> 517349 T1976 C1111 P33300 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 517350 T1974 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 517350 T1976 C1111 P33300 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 517350 T1976 C1111 P33300 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 517350 T1976 C1111 P33300 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 517351 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 517351 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 517351 T1976 C1111 P33300 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 517352 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 517356 T1921 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:33300__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 517431 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 517434 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46000
[junit4:junit4] 2> 517434 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 517435 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 517435 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130
[junit4:junit4] 2> 517435 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/solr.xml
[junit4:junit4] 2> 517436 T1883 oasc.CoreContainer.<init> New CoreContainer 1080196798
[junit4:junit4] 2> 517436 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/'
[junit4:junit4] 2> 517436 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/'
[junit4:junit4] 2> 517488 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 517488 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 517489 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 517489 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 517489 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 517490 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 517490 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 517490 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 517491 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 517491 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 517494 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 517494 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 517495 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 517496 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 517497 T1988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fb2535c name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 517498 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 517499 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 517502 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 517503 T1990 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a3ae5f9 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 517503 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 517506 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 517663 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 517664 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33300__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33300_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33300"}
[junit4:junit4] 2> 517667 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 517667 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 517667 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 517667 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 517667 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 517667 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 517667 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 518357 T1921 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:33300__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 518357 T1921 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:33300__collection1&state=recovering&nodeName=127.0.0.1:33300_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 518508 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46000_
[junit4:junit4] 2> 518510 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46000_
[junit4:junit4] 2> 518512 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 518512 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 518512 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 518512 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 518512 T1912 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518512 T1925 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518513 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 518513 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 518514 T1990 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518514 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 518514 T1905 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518515 T1941 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518515 T1957 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518515 T1973 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 518519 T1991 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 518519 T1991 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 519180 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 519181 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46000_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46000"}
[junit4:junit4] 2> 519181 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 519181 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 519186 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519186 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519186 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519186 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519186 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519186 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519186 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 519520 T1991 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/collection1
[junit4:junit4] 2> 519521 T1991 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 519522 T1991 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 519522 T1991 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 519523 T1991 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/collection1/'
[junit4:junit4] 2> 519523 T1991 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/collection1/lib/README' to classloader
[junit4:junit4] 2> 519524 T1991 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 519555 T1991 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 519581 T1991 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 519583 T1991 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 519587 T1991 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 519904 T1991 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 519909 T1991 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 519911 T1991 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 519922 T1991 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 519925 T1991 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 519927 T1991 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 519928 T1991 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 519929 T1991 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 519929 T1991 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 519930 T1991 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 519930 T1991 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 519930 T1991 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 519930 T1991 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371620931130/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/
[junit4:junit4] 2> 519931 T1991 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 519931 T1991 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 519931 T1991 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5
[junit4:junit4] 2> 519931 T1991 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index/
[junit4:junit4] 2> 519932 T1991 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 519932 T1991 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index
[junit4:junit4] 2> 519935 T1991 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 519935 T1991 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 519937 T1991 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 519937 T1991 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 519937 T1991 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 519938 T1991 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 519938 T1991 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 519938 T1991 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 519939 T1991 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 519939 T1991 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 519939 T1991 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 519942 T1991 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 519944 T1991 oass.SolrIndexSearcher.<init> Opening Searcher@22d9374c main
[junit4:junit4] 2> 519945 T1991 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 519945 T1991 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 519949 T1992 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22d9374c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 519951 T1991 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 519951 T1991 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46000 collection:collection1 shard:shard2
[junit4:junit4] 2> 519954 T1991 oasc.ZkController.register We are http://127.0.0.1:46000/collection1/ and leader is http://127.0.0.1:38379/collection1/
[junit4:junit4] 2> 519954 T1991 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46000
[junit4:junit4] 2> 519955 T1991 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 519955 T1991 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1112 name=collection1 org.apache.solr.core.SolrCore@77277fb5 url=http://127.0.0.1:46000/collection1 node=127.0.0.1:46000_ C1112_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:46000_, base_url=http://127.0.0.1:46000}
[junit4:junit4] 2> 519956 T1993 C1112 P46000 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 519956 T1993 C1112 P46000 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 519956 T1991 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 519956 T1993 C1112 P46000 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 519957 T1993 C1112 P46000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 519957 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 519958 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 519958 T1993 C1112 P46000 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 519958 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 519962 T1936 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:46000__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 520032 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 520035 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49488
[junit4:junit4] 2> 520036 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 520036 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 520036 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736
[junit4:junit4] 2> 520037 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/solr.xml
[junit4:junit4] 2> 520037 T1883 oasc.CoreContainer.<init> New CoreContainer 370436787
[junit4:junit4] 2> 520037 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/'
[junit4:junit4] 2> 520038 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/'
[junit4:junit4] 2> 520077 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 520077 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 520078 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 520078 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 520078 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 520079 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 520079 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 520079 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 520080 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 520080 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 520083 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 520083 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 520084 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 520085 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 520086 T2005 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ea87d80 name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 520087 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 520089 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 520092 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 520093 T2007 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cdb9c02 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 520093 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 520097 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1113 name=collection1 org.apache.solr.core.SolrCore@769b944a url=http://127.0.0.1:33300/collection1 node=127.0.0.1:33300_ C1113_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:33300_, base_url=http://127.0.0.1:33300}
[junit4:junit4] 2> 520358 T1976 C1113 P33300 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50699/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 520358 T1976 C1113 P33300 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33300 START replicas=[http://127.0.0.1:50699/collection1/] nUpdates=100
[junit4:junit4] 2> 520359 T1976 C1113 P33300 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 520359 T1976 C1113 P33300 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 520359 T1976 C1113 P33300 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 520360 T1976 C1113 P33300 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 520360 T1976 C1113 P33300 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 520360 T1976 C1113 P33300 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50699/collection1/. core=collection1
[junit4:junit4] 2> 520360 T1976 C1113 P33300 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1114 name=collection1 org.apache.solr.core.SolrCore@5f37bf2 url=http://127.0.0.1:50699/collection1 node=127.0.0.1:50699_ C1114_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50699_, base_url=http://127.0.0.1:50699, leader=true}
[junit4:junit4] 2> 520364 T1921 C1114 P50699 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 520368 T1920 C1114 P50699 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 520372 T1920 C1114 P50699 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 520372 T1920 C1114 P50699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 520377 T1920 C1114 P50699 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 520378 T1920 C1114 P50699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 520379 T1920 C1114 P50699 oass.SolrIndexSearcher.<init> Opening Searcher@388e753d realtime
[junit4:junit4] 2> 520379 T1920 C1114 P50699 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 520379 T1920 C1114 P50699 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 520381 T1976 C1113 P33300 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 520381 T1976 C1113 P33300 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 520383 T1919 C1114 P50699 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 520383 T1919 C1114 P50699 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 520384 T1976 C1113 P33300 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 520384 T1976 C1113 P33300 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 520384 T1976 C1113 P33300 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 520386 T1921 C1114 P50699 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 520387 T1976 C1113 P33300 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 520387 T1976 C1113 P33300 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index.20130619134854162
[junit4:junit4] 2> ASYNC NEW_CORE C1115 name=collection1 org.apache.solr.core.SolrCore@769b944a url=http://127.0.0.1:33300/collection1 node=127.0.0.1:33300_ C1115_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:33300_, base_url=http://127.0.0.1:33300}
[junit4:junit4] 2> 520394 T1976 C1115 P33300 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index.20130619134854162 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6bf38869 fullCopy=false
[junit4:junit4] 2> ASYNC NEW_CORE C1116 name=collection1 org.apache.solr.core.SolrCore@5f37bf2 url=http://127.0.0.1:50699/collection1 node=127.0.0.1:50699_ C1116_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50699_, base_url=http://127.0.0.1:50699, leader=true}
[junit4:junit4] 2> 520397 T1920 C1116 P50699 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 520399 T1976 C1115 P33300 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 520408 T1976 C1115 P33300 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 520408 T1976 C1115 P33300 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 520410 T1976 C1115 P33300 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 520411 T1976 C1115 P33300 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 520411 T1976 C1115 P33300 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 520411 T1976 C1115 P33300 oass.SolrIndexSearcher.<init> Opening Searcher@1c7b253 main
[junit4:junit4] 2> 520412 T1975 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c7b253 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 520412 T1976 C1115 P33300 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index.20130619134854162 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index.20130619134854162;done=true>>]
[junit4:junit4] 2> 520413 T1976 C1115 P33300 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index.20130619134854162
[junit4:junit4] 2> 520413 T1976 C1115 P33300 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty4/index.20130619134854162
[junit4:junit4] 2> 520413 T1976 C1115 P33300 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 520413 T1976 C1115 P33300 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 520414 T1976 C1115 P33300 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 520414 T1976 C1115 P33300 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 520416 T1976 C1115 P33300 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 520690 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 520691 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46000__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46000_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46000"}
[junit4:junit4] 2> 520693 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33300__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33300_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33300"}
[junit4:junit4] 2> 520708 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T2007 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520708 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 520963 T1936 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:46000__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 520964 T1936 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:46000__collection1&state=recovering&nodeName=127.0.0.1:46000_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 521100 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49488_
[junit4:junit4] 2> 521101 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49488_
[junit4:junit4] 2> 521103 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 521104 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 521104 T2007 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 521104 T1990 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521104 T1912 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521104 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 521104 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 521105 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 521105 T1925 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521105 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 521106 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 521107 T1905 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521107 T1957 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521107 T1973 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521108 T2007 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521108 T1941 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 521112 T2010 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 521112 T2010 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 522216 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 522217 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49488_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49488"}
[junit4:junit4] 2> 522217 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 522217 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 522221 T2007 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522222 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522222 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522222 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522222 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522222 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522221 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 522221 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> ASYNC NEW_CORE C1117 name=collection1 org.apache.solr.core.SolrCore@77277fb5 url=http://127.0.0.1:46000/collection1 node=127.0.0.1:46000_ C1117_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:46000_, base_url=http://127.0.0.1:46000}
[junit4:junit4] 2> 522964 T1993 C1117 P46000 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38379/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 522965 T1993 C1117 P46000 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46000 START replicas=[http://127.0.0.1:38379/collection1/] nUpdates=100
[junit4:junit4] 2> 522965 T1993 C1117 P46000 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 522966 T1993 C1117 P46000 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 522966 T1993 C1117 P46000 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 522967 T1993 C1117 P46000 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 522967 T1993 C1117 P46000 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 522967 T1993 C1117 P46000 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38379/collection1/. core=collection1
[junit4:junit4] 2> 522968 T1993 C1117 P46000 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1118 name=collection1 org.apache.solr.core.SolrCore@6c6c80b8 url=http://127.0.0.1:38379/collection1 node=127.0.0.1:38379_ C1118_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38379_, base_url=http://127.0.0.1:38379, leader=true}
[junit4:junit4] 2> 522972 T1935 C1118 P38379 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 522974 T1934 C1118 P38379 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 522979 T1934 C1118 P38379 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 522979 T1934 C1118 P38379 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 522997 T1934 C1118 P38379 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 522998 T1934 C1118 P38379 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 522999 T1934 C1118 P38379 oass.SolrIndexSearcher.<init> Opening Searcher@f7ef579 realtime
[junit4:junit4] 2> 522999 T1934 C1118 P38379 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 523000 T1934 C1118 P38379 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 26
[junit4:junit4] 2> 523001 T1993 C1117 P46000 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 523001 T1993 C1117 P46000 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 523002 T1935 C1118 P38379 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 523003 T1935 C1118 P38379 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 523003 T1993 C1117 P46000 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 523003 T1993 C1117 P46000 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 523004 T1993 C1117 P46000 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 523006 T1937 C1118 P38379 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 523006 T1993 C1117 P46000 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 523007 T1993 C1117 P46000 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index.20130619134856781
[junit4:junit4] 2> 523007 T1993 C1117 P46000 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index.20130619134856781 lockFactory=org.apache.lucene.store.NativeFSLockFactory@fad3357 fullCopy=false
[junit4:junit4] 2> 523009 T1934 C1118 P38379 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 523011 T1993 C1117 P46000 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 523018 T1993 C1117 P46000 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 523018 T1993 C1117 P46000 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 523020 T1993 C1117 P46000 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 523020 T1993 C1117 P46000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 523020 T1993 C1117 P46000 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 523021 T1993 C1117 P46000 oass.SolrIndexSearcher.<init> Opening Searcher@66fc8fdc main
[junit4:junit4] 2> 523021 T1992 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66fc8fdc main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 523022 T1993 C1117 P46000 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index.20130619134856781 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index.20130619134856781;done=true>>]
[junit4:junit4] 2> 523022 T1993 C1117 P46000 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index.20130619134856781
[junit4:junit4] 2> 523022 T1993 C1117 P46000 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty5/index.20130619134856781
[junit4:junit4] 2> 523023 T1993 C1117 P46000 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 523023 T1993 C1117 P46000 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 523023 T1993 C1117 P46000 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 523023 T1993 C1117 P46000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 523025 T1993 C1117 P46000 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 523113 T2010 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/collection1
[junit4:junit4] 2> 523114 T2010 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 523114 T2010 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 523115 T2010 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 523116 T2010 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/collection1/'
[junit4:junit4] 2> 523116 T2010 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/collection1/lib/README' to classloader
[junit4:junit4] 2> 523116 T2010 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 523145 T2010 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 523182 T2010 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 523183 T2010 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 523188 T2010 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 523540 T2010 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 523546 T2010 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 523548 T2010 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 523562 T2010 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 523566 T2010 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 523570 T2010 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 523571 T2010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 523572 T2010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 523572 T2010 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 523573 T2010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 523573 T2010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 523573 T2010 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 523574 T2010 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371620933736/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty6/
[junit4:junit4] 2> 523574 T2010 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 523574 T2010 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 523575 T2010 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty6
[junit4:junit4] 2> 523575 T2010 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty6/index/
[junit4:junit4] 2> 523576 T2010 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 523576 T2010 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty6/index
[junit4:junit4] 2> 523580 T2010 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 523580 T2010 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 523582 T2010 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 523583 T2010 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 523583 T2010 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 523584 T2010 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 523585 T2010 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 523585 T2010 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 523588 T2010 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 523589 T2010 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 523589 T2010 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 523594 T2010 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 523599 T2010 oass.SolrIndexSearcher.<init> Opening Searcher@75895466 main
[junit4:junit4] 2> 523600 T2010 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 523600 T2010 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 523604 T2013 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@75895466 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 523606 T2010 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 523607 T2010 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49488 collection:collection1 shard:shard3
[junit4:junit4] 2> 523612 T2010 oasc.ZkController.register We are http://127.0.0.1:49488/collection1/ and leader is http://127.0.0.1:55543/collection1/
[junit4:junit4] 2> 523612 T2010 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49488
[junit4:junit4] 2> 523612 T2010 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 523612 T2010 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1119 name=collection1 org.apache.solr.core.SolrCore@46c96ab3 url=http://127.0.0.1:49488/collection1 node=127.0.0.1:49488_ C1119_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49488_, base_url=http://127.0.0.1:49488}
[junit4:junit4] 2> 523613 T2014 C1119 P49488 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 523615 T2010 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 523617 T2014 C1119 P49488 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 523619 T1883 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 523620 T2014 C1119 P49488 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 523621 T2014 C1119 P49488 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 523620 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 523622 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 523624 T2014 C1119 P49488 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 523630 T1951 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:49488__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 523727 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 523728 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46000__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46000_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46000"}
[junit4:junit4] 2> 523731 T1883 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 523731 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49488__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49488_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49488"}
[junit4:junit4] 2> 523734 T1883 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52277
[junit4:junit4] 2> 523735 T1883 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 523735 T1883 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 523736 T1883 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402
[junit4:junit4] 2> 523736 T1883 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/solr.xml
[junit4:junit4] 2> 523736 T1883 oasc.CoreContainer.<init> New CoreContainer 1818300531
[junit4:junit4] 2> 523737 T1883 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/'
[junit4:junit4] 2> 523737 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/'
[junit4:junit4] 2> 523741 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523742 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523742 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523742 T2007 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523741 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523742 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523742 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523742 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 523819 T1883 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 523820 T1883 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 523820 T1883 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 523821 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 523821 T1883 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 523822 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 523823 T1883 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 523823 T1883 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 523824 T1883 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 523824 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 523830 T1883 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 523831 T1883 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42979/solr
[junit4:junit4] 2> 523831 T1883 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 523832 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 523834 T2026 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55138a66 name:ZooKeeperConnection Watcher:127.0.0.1:42979 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 523839 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 523841 T1883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 523848 T1883 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 523849 T2028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f7c9c00 name:ZooKeeperConnection Watcher:127.0.0.1:42979/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 523850 T1883 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 523863 T1883 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 524631 T1951 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:49488__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 524631 T1951 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:49488__collection1&state=recovering&nodeName=127.0.0.1:49488_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 524869 T1883 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52277_
[junit4:junit4] 2> 524870 T1883 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52277_
[junit4:junit4] 2> 524873 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 524873 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 524873 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 524874 T1912 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524874 T1925 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524873 T2007 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 524873 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 524874 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 524874 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 524874 T1990 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524876 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 524877 T1941 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524879 T1905 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524879 T2028 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524880 T1973 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524880 T2007 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524881 T1957 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 524888 T2029 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 524889 T2029 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 525250 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 525250 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52277_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52277"}
[junit4:junit4] 2> 525251 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 525251 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 525318 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T1941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T1990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T1925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T1973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T1957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T2028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T1912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525318 T2007 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 525890 T2029 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/collection1
[junit4:junit4] 2> 525891 T2029 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 525892 T2029 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 525892 T2029 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 525893 T2029 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/collection1/'
[junit4:junit4] 2> 525894 T2029 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/collection1/lib/README' to classloader
[junit4:junit4] 2> 525894 T2029 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 525945 T2029 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 525990 T2029 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 525991 T2029 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 525997 T2029 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 526542 T2029 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 526549 T2029 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 526552 T2029 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 526564 T2029 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 526569 T2029 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 526575 T2029 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 526576 T2029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 526577 T2029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 526577 T2029 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 526579 T2029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 526579 T2029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 526580 T2029 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 526580 T2029 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371620937402/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty7/
[junit4:junit4] 2> 526580 T2029 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e3d101
[junit4:junit4] 2> 526581 T2029 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 526582 T2029 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371620915984/jetty7
[junit4:junit4] 2> 526582 T2029 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud
[...truncated too long message...]
ection",
[junit4:junit4] 1> "node_name":"127.0.0.1:58486__%2Fm",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:58486/_/m",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:55790__%2Fm (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:58486__%2Fm (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:56644__%2Fm (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46392__%2Fm (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890552449859594-127.0.0.1:46392__%2Fm_collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890552449859598-127.0.0.1:55790__%2Fm_collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89890552449859596-127.0.0.1:56644__%2Fm_collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:46392__%2Fm",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46392/_/m"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89890552449859587-127.0.0.1:58486__%2Fm_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:58486__%2Fm",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:58486/_/m"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89890552449859594-127.0.0.1:46392__%2Fm-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890552449859587-127.0.0.1:58486__%2Fm-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890552449859598-127.0.0.1:55790__%2Fm-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89890552449859596-127.0.0.1:56644__%2Fm-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89890552449859587-127.0.0.1:58486__%2Fm-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=DBD5C3FDF836292C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hi_IN -Dtests.timezone=SystemV/EST5 -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 102s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:55790/_/m/collection1lastClient and got 250 from http://127.0.0.1:46392/_/m/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([DBD5C3FDF836292C:5A334DE58F694910]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 665842 T2336 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 102018 T2335 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 667030 T2428 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 667031 T2428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 667031 T2428 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=TestBloomFilteredLucene41Postings), _version_=PostingsFormat(name=Memory doPackFST= false), rnd_b=Pulsing41(freqCutoff=3 minBlockSize=69 maxBlockSize=198), intDefault=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false), id=Pulsing41(freqCutoff=3 minBlockSize=69 maxBlockSize=198), a_t=PostingsFormat(name=Memory doPackFST= false), range_facet_sl=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_si=PostingsFormat(name=MockRandom), other_tl1=PostingsFormat(name=Memory doPackFST= false), multiDefault=PostingsFormat(name=MockRandom), a_si=PostingsFormat(name=MockRandom)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=hi_IN, timezone=SystemV/EST5
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=3,free=180052872,total=313454592
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestLRUCache, SystemInfoHandlerTest, LegacyHTMLStripCharFilterTest, PluginInfoTest, DateFieldTest, ChaosMonkeyNothingIsSafeTest, TestManagedSchema, RecoveryZkTest, FieldMutatingUpdateProcessorTest, PeerSyncTest, LoggingHandlerTest, TestSweetSpotSimilarityFactory, SuggesterTest, TestFastLRUCache, TestSort, OverseerTest, TestJoin, TestValueSourceCache, TestFunctionQuery, TestSolrCoreProperties, DirectSolrConnectionTest, TestCharFilters, TestExtendedDismaxParser, PreAnalyzedUpdateProcessorTest, TestSolrDeletionPolicy2, DocumentBuilderTest, ReturnFieldsTest, PrimUtilsTest, LukeRequestHandlerTest, SOLR749Test, UpdateRequestProcessorFactoryTest, TestFieldCollectionResource, TestFastWriter, TestSchemaSimilarityResource, ZkCLITest, TestQuerySenderListener, DistributedTermsComponentTest, IndexSchemaTest, TestPropInjectDefaults, TestCoreContainer, TestPerFieldSimilarity, QueryEqualityTest, SynonymTokenizerTest, JsonLoaderTest, UnloadDistributedZkTest, RegexBoostProcessorTest, ExternalFileFieldSortTest, TestReloadAndDeleteDocs, SolrIndexSplitterTest, SolrCoreTest, TestLFUCache, SimpleFacetsTest, PrimitiveFieldTypeTest, OverseerCollectionProcessorTest, TestCloudManagedSchemaAddField, DocValuesTest, ShowFileRequestHandlerTest, TestLMDirichletSimilarityFactory, TestRealTimeGet, ResponseLogComponentTest, TestSchemaResource, AlternateDirectoryTest, WordBreakSolrSpellCheckerTest, SoftAutoCommitTest, TestSolrXmlPersistence, TestSolrIndexConfig, SuggesterFSTTest, QueryParsingTest, TestGroupingSearch, PreAnalyzedFieldTest, TestQueryUtils, SearchHandlerTest, ScriptEngineTest, BasicDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 103.29s, 1 test, 1 failure <<< FAILURES!
[...truncated 460 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1252 tests, 2 failures, 13 ignored (7 assumptions)
Total time: 38 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6203 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6203/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<184> but was:<32>
Stack Trace:
java.lang.AssertionError: expected:<184> but was:<32>
at __randomizedtesting.SeedInfo.seed([51B29FEFC84DEB49:D05411F7BF128B75]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9335 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 236228 T984 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /si_/
[junit4:junit4] 2> 236234 T984 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371615470614
[junit4:junit4] 2> 236234 T984 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 236235 T985 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 236335 T984 oasc.ZkTestServer.run start zk server on port:46157
[junit4:junit4] 2> 236336 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 236339 T991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43e5de04 name:ZooKeeperConnection Watcher:127.0.0.1:46157 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 236340 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 236340 T984 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 236344 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 236345 T993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57312a4b name:ZooKeeperConnection Watcher:127.0.0.1:46157/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 236346 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 236346 T984 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 236349 T984 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 236352 T984 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 236354 T984 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 236356 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 236356 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 236360 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 236360 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 236363 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 236363 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 236365 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 236366 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 236368 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 236368 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 236370 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 236371 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 236373 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 236373 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 236375 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 236376 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 236378 T984 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 236378 T984 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 236472 T984 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 236475 T984 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33035
[junit4:junit4] 2> 236476 T984 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 236476 T984 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 236476 T984 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761
[junit4:junit4] 2> 236477 T984 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/solr.xml
[junit4:junit4] 2> 236477 T984 oasc.CoreContainer.<init> New CoreContainer 1455914138
[junit4:junit4] 2> 236478 T984 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/'
[junit4:junit4] 2> 236478 T984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/'
[junit4:junit4] 2> 236524 T984 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 236524 T984 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 236525 T984 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 236525 T984 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 236525 T984 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 236526 T984 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 236526 T984 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 236526 T984 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 236527 T984 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 236527 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 236530 T984 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 236530 T984 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46157/solr
[junit4:junit4] 2> 236530 T984 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 236531 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 236533 T1004 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd9613 name:ZooKeeperConnection Watcher:127.0.0.1:46157 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 236534 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 236536 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 236538 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 236540 T1006 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@173883f4 name:ZooKeeperConnection Watcher:127.0.0.1:46157/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 236540 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 236541 T984 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 236544 T984 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 236547 T984 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 236548 T984 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33035_si_
[junit4:junit4] 2> 236549 T984 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33035_si_
[junit4:junit4] 2> 236552 T984 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 236556 T984 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 236558 T984 oasc.Overseer.start Overseer (id=89890191482290179-127.0.0.1:33035_si_-n_0000000000) starting
[junit4:junit4] 2> 236561 T984 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 236565 T1008 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 236565 T984 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 236568 T984 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 236570 T984 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 236572 T1007 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 236574 T1009 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 236574 T1009 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 238075 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 238076 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33035_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33035/si_"}
[junit4:junit4] 2> 238076 T1007 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 238076 T1007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 238079 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 238575 T1009 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/collection1
[junit4:junit4] 2> 238576 T1009 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 238577 T1009 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 238577 T1009 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 238578 T1009 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/collection1/'
[junit4:junit4] 2> 238579 T1009 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/collection1/lib/README' to classloader
[junit4:junit4] 2> 238579 T1009 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 238611 T1009 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 238652 T1009 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 238653 T1009 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 238658 T1009 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 239023 T1009 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 239030 T1009 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 239032 T1009 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 239054 T1009 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 239059 T1009 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 239064 T1009 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 239065 T1009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 239066 T1009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 239066 T1009 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 239067 T1009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 239067 T1009 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 239068 T1009 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 239068 T1009 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371615470761/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/control/data/
[junit4:junit4] 2> 239068 T1009 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@429d486d
[junit4:junit4] 2> 239069 T1009 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 239070 T1009 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/control/data
[junit4:junit4] 2> 239070 T1009 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/control/data/index/
[junit4:junit4] 2> 239070 T1009 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371615470614/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 239071 T1009 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/control/data/index
[junit4:junit4] 2> 239073 T1009 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2cecec49 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2db67e7c)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 239073 T1009 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 239076 T1009 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 239076 T1009 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 239077 T1009 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 239077 T1009 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 239078 T1009 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 239078 T1009 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 239078 T1009 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 239079 T1009 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 239079 T1009 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 239083 T1009 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 239087 T1009 oass.SolrIndexSearcher.<init> Opening Searcher@289d184b main
[junit4:junit4] 2> 239088 T1009 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/control/data/tlog
[junit4:junit4] 2> 239088 T1009 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 239089 T1009 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 239092 T1010 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@289d184b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 239093 T1009 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 239093 T1009 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33035/si_ collection:control_collection shard:shard1
[junit4:junit4] 2> 239094 T1009 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 239099 T1009 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 239102 T1009 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 239102 T1009 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 239102 T1009 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33035/si_/collection1/
[junit4:junit4] 2> 239102 T1009 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 239102 T1009 oasc.SyncStrategy.syncToMe http://127.0.0.1:33035/si_/collection1/ has no replicas
[junit4:junit4] 2> 239103 T1009 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33035/si_/collection1/
[junit4:junit4] 2> 239103 T1009 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 239582 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 239588 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 239610 T1009 oasc.ZkController.register We are http://127.0.0.1:33035/si_/collection1/ and leader is http://127.0.0.1:33035/si_/collection1/
[junit4:junit4] 2> 239610 T1009 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33035/si_
[junit4:junit4] 2> 239610 T1009 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 239610 T1009 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 239610 T1009 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 239612 T1009 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 239612 T984 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 239613 T984 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 239613 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 239617 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 239618 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 239619 T1013 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@449d2f28 name:ZooKeeperConnection Watcher:127.0.0.1:46157/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 239619 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 239621 T984 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 239622 T984 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 239692 T984 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 239694 T984 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57946
[junit4:junit4] 2> 239694 T984 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 239694 T984 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 239695 T984 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002
[junit4:junit4] 2> 239695 T984 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/solr.xml
[junit4:junit4] 2> 239695 T984 oasc.CoreContainer.<init> New CoreContainer 63946807
[junit4:junit4] 2> 239696 T984 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/'
[junit4:junit4] 2> 239696 T984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/'
[junit4:junit4] 2> 239734 T984 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 239734 T984 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 239734 T984 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 239735 T984 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 239735 T984 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 239735 T984 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 239735 T984 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 239736 T984 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 239736 T984 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 239736 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 239739 T984 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 239739 T984 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46157/solr
[junit4:junit4] 2> 239740 T984 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 239740 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 239742 T1024 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a207ddd name:ZooKeeperConnection Watcher:127.0.0.1:46157 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 239742 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 239744 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 239747 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 239748 T1026 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@361578f1 name:ZooKeeperConnection Watcher:127.0.0.1:46157/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 239748 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 239751 T984 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 240754 T984 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57946_si_
[junit4:junit4] 2> 240755 T984 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57946_si_
[junit4:junit4] 2> 240758 T1006 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 240758 T1013 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 240758 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 240758 T1026 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 240769 T1027 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 240769 T1027 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 241092 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 241093 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33035_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33035/si_"}
[junit4:junit4] 2> 241096 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57946_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57946/si_"}
[junit4:junit4] 2> 241096 T1007 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 241096 T1007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 241101 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 241101 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 241101 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 241771 T1027 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/collection1
[junit4:junit4] 2> 241771 T1027 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 241772 T1027 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 241772 T1027 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 241773 T1027 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/collection1/'
[junit4:junit4] 2> 241773 T1027 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/collection1/lib/README' to classloader
[junit4:junit4] 2> 241774 T1027 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 241799 T1027 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 241833 T1027 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 241834 T1027 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 241838 T1027 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 242150 T1027 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 242155 T1027 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 242157 T1027 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 242171 T1027 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 242175 T1027 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 242179 T1027 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 242181 T1027 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 242181 T1027 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 242181 T1027 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 242182 T1027 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 242182 T1027 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 242183 T1027 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 242183 T1027 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371615474002/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty1/
[junit4:junit4] 2> 242183 T1027 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@429d486d
[junit4:junit4] 2> 242184 T1027 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 242185 T1027 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty1
[junit4:junit4] 2> 242185 T1027 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty1/index/
[junit4:junit4] 2> 242185 T1027 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 242186 T1027 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty1/index
[junit4:junit4] 2> 242188 T1027 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@53e3b7f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a6e08a4)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 242188 T1027 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 242190 T1027 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 242191 T1027 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 242191 T1027 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 242192 T1027 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 242192 T1027 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 242193 T1027 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 242193 T1027 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 242194 T1027 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 242194 T1027 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 242197 T1027 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 242199 T1027 oass.SolrIndexSearcher.<init> Opening Searcher@73a26d7c main
[junit4:junit4] 2> 242200 T1027 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty1/tlog
[junit4:junit4] 2> 242200 T1027 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 242200 T1027 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 242204 T1028 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73a26d7c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 242206 T1027 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 242206 T1027 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57946/si_ collection:collection1 shard:shard1
[junit4:junit4] 2> 242207 T1027 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 242212 T1027 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 242214 T1027 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 242214 T1027 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 242214 T1027 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57946/si_/collection1/
[junit4:junit4] 2> 242215 T1027 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 242215 T1027 oasc.SyncStrategy.syncToMe http://127.0.0.1:57946/si_/collection1/ has no replicas
[junit4:junit4] 2> 242215 T1027 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57946/si_/collection1/
[junit4:junit4] 2> 242215 T1027 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 242608 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 242615 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 242615 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 242615 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 242621 T1027 oasc.ZkController.register We are http://127.0.0.1:57946/si_/collection1/ and leader is http://127.0.0.1:57946/si_/collection1/
[junit4:junit4] 2> 242621 T1027 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57946/si_
[junit4:junit4] 2> 242621 T1027 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 242622 T1027 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 242622 T1027 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 242624 T1027 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 242625 T984 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 242625 T984 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 242625 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 242693 T984 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 242695 T984 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41032
[junit4:junit4] 2> 242695 T984 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 242696 T984 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 242696 T984 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008
[junit4:junit4] 2> 242696 T984 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/solr.xml
[junit4:junit4] 2> 242697 T984 oasc.CoreContainer.<init> New CoreContainer 1951691451
[junit4:junit4] 2> 242697 T984 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/'
[junit4:junit4] 2> 242697 T984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/'
[junit4:junit4] 2> 242744 T984 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 242745 T984 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 242745 T984 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 242745 T984 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 242745 T984 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 242746 T984 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 242746 T984 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 242746 T984 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 242747 T984 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 242747 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 242750 T984 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 242750 T984 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46157/solr
[junit4:junit4] 2> 242751 T984 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 242751 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 242753 T1040 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@299430f6 name:ZooKeeperConnection Watcher:127.0.0.1:46157 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 242753 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 242755 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 242757 T984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 242758 T1042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ffd024f name:ZooKeeperConnection Watcher:127.0.0.1:46157/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 242758 T984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 242761 T984 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 243763 T984 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41032_si_
[junit4:junit4] 2> 243764 T984 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41032_si_
[junit4:junit4] 2> 243767 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 243768 T1006 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 243768 T1013 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 243768 T1042 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 243768 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 243768 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 243769 T1026 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 243774 T1043 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 243774 T1043 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 244119 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 244120 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57946_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57946/si_"}
[junit4:junit4] 2> 244124 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41032_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41032/si_"}
[junit4:junit4] 2> 244124 T1007 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 244124 T1007 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 244128 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 244128 T1042 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 244128 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 244128 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 244775 T1043 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/collection1
[junit4:junit4] 2> 244776 T1043 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 244777 T1043 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 244777 T1043 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 244778 T1043 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/collection1/'
[junit4:junit4] 2> 244779 T1043 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/collection1/lib/README' to classloader
[junit4:junit4] 2> 244779 T1043 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 244818 T1043 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 244851 T1043 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 244853 T1043 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 244857 T1043 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 245183 T1043 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 245188 T1043 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 245190 T1043 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 245203 T1043 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 245207 T1043 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 245210 T1043 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 245211 T1043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 245212 T1043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 245212 T1043 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 245213 T1043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 245213 T1043 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 245213 T1043 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 245213 T1043 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371615477008/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/
[junit4:junit4] 2> 245214 T1043 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@429d486d
[junit4:junit4] 2> 245214 T1043 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 245214 T1043 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2
[junit4:junit4] 2> 245215 T1043 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index/
[junit4:junit4] 2> 245215 T1043 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 245215 T1043 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index
[junit4:junit4] 2> 245216 T1043 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 245216 T1043 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 245218 T1043 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 245218 T1043 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 245219 T1043 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 245219 T1043 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 245219 T1043 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 245220 T1043 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 245220 T1043 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 245220 T1043 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 245220 T1043 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 245223 T1043 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 245225 T1043 oass.SolrIndexSearcher.<init> Opening Searcher@15f9db52 main
[junit4:junit4] 2> 245225 T1043 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/tlog
[junit4:junit4] 2> 245225 T1043 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 245226 T1043 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 245229 T1044 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15f9db52 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 245230 T1043 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 245230 T1043 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41032/si_ collection:collection1 shard:shard1
[junit4:junit4] 2> 245233 T1043 oasc.ZkController.register We are http://127.0.0.1:41032/si_/collection1/ and leader is http://127.0.0.1:57946/si_/collection1/
[junit4:junit4] 2> 245233 T1043 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41032/si_
[junit4:junit4] 2> 245233 T1043 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 245233 T1043 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1424 name=collection1 org.apache.solr.core.SolrCore@4baeebfd url=http://127.0.0.1:41032/si_/collection1 node=127.0.0.1:41032_si_ C1424_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41032_si_, base_url=http://127.0.0.1:41032/si_}
[junit4:junit4] 2> 245233 T1045 C1424 P41032 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 245234 T1043 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 245234 T1045 C1424 P41032 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 245234 T1045 C1424 P41032 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 245234 T1045 C1424 P41032 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 245235 T984 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 245235 T984 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 245235 T984 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 245238 T1045 C1424 P41032 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 245238 T984 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 245242 T1022 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C1425 name=collection1 org.apache.solr.core.SolrCore@20be20df url=http://127.0.0.1:33035/si_/collection1 node=127.0.0.1:33035_si_ C1425_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:33035_si_, base_url=http://127.0.0.1:33035/si_, leader=true}
[junit4:junit4] 2> 245249 T1000 C1425 P33035 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2cecec49 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2db67e7c)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 245249 T1000 C1425 P33035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 245252 T1001 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10001 (1438243073165361152)]} 0 5
[junit4:junit4] 2> 245252 T1000 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[1 (1438243073165361153)]} 0 5
[junit4:junit4] 2> ASYNC NEW_CORE C1426 name=collection1 org.apache.solr.core.SolrCore@4663c5fd url=http://127.0.0.1:57946/si_/collection1 node=127.0.0.1:57946_si_ C1426_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57946_si_, base_url=http://127.0.0.1:57946/si_, leader=true}
[junit4:junit4] 2> 245261 T1020 C1426 P57946 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@53e3b7f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a6e08a4)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 245262 T1020 C1426 P57946 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 245264 T1020 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10001 (1438243073177944064)]} 0 5
[junit4:junit4] 2> 245265 T1021 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[1 (1438243073180041216)]} 0 4
[junit4:junit4] 2> 245325 T1001 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[2 (1438243073246101505)]} 0 1
[junit4:junit4] 2> 245325 T1002 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10002 (1438243073246101504)]} 0 1
[junit4:junit4] 2> 245329 T1020 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10002 (1438243073250295809)]} 0 1
[junit4:junit4] 2> 245329 T1019 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[2 (1438243073250295808)]} 0 1
[junit4:junit4] 2> 245425 T999 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10002 (-1438243073352007680)]} 0 1
[junit4:junit4] 2> 245425 T1000 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[2 (-1438243073352007681)]} 0 1
[junit4:junit4] 2> 245428 T1021 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10002 (-1438243073355153408)]} 0 0
[junit4:junit4] 2> 245428 T1020 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[2 (-1438243073355153409)]} 0 0
[junit4:junit4] 2> 245431 T1001 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10003 (1438243073357250560)]} 0 1
[junit4:junit4] 2> 245431 T1002 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[3 (1438243073357250561)]} 0 1
[junit4:junit4] 2> 245434 T1019 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10003 (1438243073360396288)]} 0 1
[junit4:junit4] 2> 245434 T1021 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[3 (1438243073360396289)]} 0 1
[junit4:junit4] 2> 245440 T984 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 41032
[junit4:junit4] 2> 245440 T984 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1951691451
[junit4:junit4] 2> 245535 T999 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10004 (1438243073466302464)]} 0 1
[junit4:junit4] 2> 245536 T1000 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[4 (1438243073467351040)]} 0 2
[junit4:junit4] 2> 245539 T1020 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10004 (1438243073470496768)]} 0 1
[junit4:junit4] 2> 245539 T1019 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[4 (1438243073470496769)]} 0 1
[junit4:junit4] 2> 245604 T1001 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10005 (1438243073538654208)]} 0 1
[junit4:junit4] 2> 245604 T1002 C1425 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[5 (1438243073538654209)]} 0 1
[junit4:junit4] 2> 245607 T1021 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10005 (1438243073541799936)]} 0 1
[junit4:junit4] 2> 245608 T1020 C1426 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[5 (1438243073541799937)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C1427 name=collection1 org.apache.solr.core.SolrCore@20be20df url=http://127.0.0.1:33035/si_/collection1 node=127.0.0.1:33035_si_ C1427_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:33035_si_, base_url=http://127.0.0.1:33035/si_, leader=true}
[junit4:junit4] 2> 245637 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10006 (1438243073574305792)]} 0 1
[junit4:junit4] 2> 245638 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[6 (1438243073574305793)]} 0 2
[junit4:junit4] 2> 245638 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 245639 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41032_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41032/si_"}
[junit4:junit4] 2> ASYNC NEW_CORE C1428 name=collection1 org.apache.solr.core.SolrCore@4663c5fd url=http://127.0.0.1:57946/si_/collection1 node=127.0.0.1:57946_si_ C1428_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57946_si_, base_url=http://127.0.0.1:57946/si_, leader=true}
[junit4:junit4] 2> 245641 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10006 (1438243073577451520)]} 0 1
[junit4:junit4] 2> 245641 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[6 (1438243073577451521)]} 0 1
[junit4:junit4] 2> 245643 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 245643 T1042 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 245643 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 245643 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 245722 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10005 (-1438243073663434752)]} 0 1
[junit4:junit4] 2> 245722 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[5 (-1438243073663434753)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C1429 name=collection1 org.apache.solr.core.SolrCore@4baeebfd url=http://127.0.0.1:41032/si_/collection1 node=127.0.0.1:41032_si_ C1429_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41032_si_, base_url=http://127.0.0.1:41032/si_}
[junit4:junit4] 2> 245731 T1035 C1429 P41032 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 245732 T1035 C1429 P41032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 245732 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10005 (-1438243073665531904)]} 0 3
[junit4:junit4] 2> 245732 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[5 (-1438243073665531905)]} 0 1
[junit4:junit4] 2> 245733 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10005 (-1438243073665531904)]} 0 9
[junit4:junit4] 2> 245733 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[5 (-1438243073665531905)]} 0 9
[junit4:junit4] 2> 245736 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[7 (1438243073678114816)]} 0 1
[junit4:junit4] 2> 245736 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10007 (1438243073677066240)]} 0 1
[junit4:junit4] 2> 245742 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1438243073680211969)]} 0 1
[junit4:junit4] 2> 245743 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[7 (1438243073680211969)]} 0 5
[junit4:junit4] 2> 245743 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10007 (1438243073680211968)]} 0 2
[junit4:junit4] 2> 245744 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10007 (1438243073680211968)]} 0 6
[junit4:junit4] 2> 245763 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[8 (1438243073705377792)]} 0 2
[junit4:junit4] 2> 245763 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10008 (1438243073705377793)]} 0 1
[junit4:junit4] 2> 245768 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1438243073708523520)]} 0 1
[junit4:junit4] 2> 245769 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[8 (1438243073708523520)]} 0 4
[junit4:junit4] 2> 245769 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10008 (1438243073708523521)]} 0 1
[junit4:junit4] 2> 245770 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10008 (1438243073708523521)]} 0 5
[junit4:junit4] 2> 245810 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[9 (1438243073754660864)]} 0 1
[junit4:junit4] 2> 245810 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10009 (1438243073755709440)]} 0 0
[junit4:junit4] 2> 245815 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1438243073757806592)]} 0 0
[junit4:junit4] 2> 245816 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[9 (1438243073757806592)]} 0 4
[junit4:junit4] 2> 245816 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10009 (1438243073757806593)]} 0 1
[junit4:junit4] 2> 245817 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10009 (1438243073757806593)]} 0 5
[junit4:junit4] 2> 245841 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[7 (-1438243073788215296)]} 0 0
[junit4:junit4] 2> 245842 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10007 (-1438243073789263872)]} 0 0
[junit4:junit4] 2> 245846 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[7 (-1438243073791361024)]} 0 0
[junit4:junit4] 2> 245846 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10007 (-1438243073791361025)]} 0 0
[junit4:junit4] 2> 245846 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[7 (-1438243073791361024)]} 0 2
[junit4:junit4] 2> 245847 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10007 (-1438243073791361025)]} 0 3
[junit4:junit4] 2> 245849 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10 (1438243073796603904)]} 0 1
[junit4:junit4] 2> 245849 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10010 (1438243073796603905)]} 0 0
[junit4:junit4] 2> 245854 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1438243073798701056)]} 0 0
[junit4:junit4] 2> 245855 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10 (1438243073798701056)]} 0 4
[junit4:junit4] 2> 245855 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10010 (1438243073799749632)]} 0 1
[junit4:junit4] 2> 245856 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10010 (1438243073799749632)]} 0 5
[junit4:junit4] 2> 245916 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10 (-1438243073866858496)]} 0 0
[junit4:junit4] 2> 245917 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10010 (-1438243073867907072)]} 0 0
[junit4:junit4] 2> 245921 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10 (-1438243073870004224)]} 0 0
[junit4:junit4] 2> 245921 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10010 (-1438243073870004225)]} 0 0
[junit4:junit4] 2> 245921 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10 (-1438243073870004224)]} 0 2
[junit4:junit4] 2> 245922 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10010 (-1438243073870004225)]} 0 3
[junit4:junit4] 2> 245925 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[11 (1438243073875247104)]} 0 1
[junit4:junit4] 2> 245925 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10011 (1438243073875247105)]} 0 1
[junit4:junit4] 2> 245931 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1438243073878392832)]} 0 2
[junit4:junit4] 2> 245931 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10011 (1438243073878392833)]} 0 1
[junit4:junit4] 2> 245931 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[11 (1438243073878392832)]} 0 4
[junit4:junit4] 2> 245931 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10011 (1438243073878392833)]} 0 4
[junit4:junit4] 2> 245961 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[12 (1438243073912995841)]} 0 1
[junit4:junit4] 2> 245961 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10012 (1438243073912995840)]} 0 1
[junit4:junit4] 2> 245967 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1438243073917190144)]} 0 1
[junit4:junit4] 2> 245968 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[12 (1438243073917190144)]} 0 5
[junit4:junit4] 2> 245968 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10012 (1438243073917190145)]} 0 1
[junit4:junit4] 2> 245968 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10012 (1438243073917190145)]} 0 5
[junit4:junit4] 2> 245990 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[13 (1438243073943404544)]} 0 1
[junit4:junit4] 2> 245990 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10013 (1438243073944453120)]} 0 1
[junit4:junit4] 2> 245996 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1438243073947598848)]} 0 1
[junit4:junit4] 2> 245996 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10013 (1438243073947598849)]} 0 1
[junit4:junit4] 2> 245997 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[13 (1438243073947598848)]} 0 5
[junit4:junit4] 2> 245997 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10013 (1438243073947598849)]} 0 4
[junit4:junit4] 2> 246036 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[14 (1438243073991639041)]} 0 1
[junit4:junit4] 2> 246037 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10014 (1438243073991639040)]} 0 2
[junit4:junit4] 2> 246042 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1438243073995833345)]} 0 1
[junit4:junit4] 2> 246043 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1438243073995833344)]} 0 1
[junit4:junit4] 2> 246043 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10014 (1438243073995833345)]} 0 4
[junit4:junit4] 2> 246043 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[14 (1438243073995833344)]} 0 4
[junit4:junit4] 2> 246087 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[15 (1438243074045116417)]} 0 1
[junit4:junit4] 2> 246087 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10015 (1438243074045116416)]} 0 1
[junit4:junit4] 2> 246093 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1438243074048262144)]} 0 1
[junit4:junit4] 2> 246094 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1438243074048262145)]} 0 2
[junit4:junit4] 2> 246094 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10015 (1438243074048262144)]} 0 5
[junit4:junit4] 2> 246094 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[15 (1438243074048262145)]} 0 5
[junit4:junit4] 2> 246157 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10011 (-1438243074119565312)]} 0 1
[junit4:junit4] 2> 246157 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[11 (-1438243074119565313)]} 0 1
[junit4:junit4] 2> 246162 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10011 (-1438243074122711041)]} 0 0
[junit4:junit4] 2> 246163 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[11 (-1438243074122711040)]} 0 1
[junit4:junit4] 2> 246163 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10011 (-1438243074122711041)]} 0 4
[junit4:junit4] 2> 246163 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[11 (-1438243074122711040)]} 0 4
[junit4:junit4] 2> 246166 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10016 (1438243074127953920)]} 0 1
[junit4:junit4] 2> 246166 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[16 (1438243074127953921)]} 0 1
[junit4:junit4] 2> 246171 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1438243074131099648)]} 0 1
[junit4:junit4] 2> 246172 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1438243074131099649)]} 0 1
[junit4:junit4] 2> 246172 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10016 (1438243074131099648)]} 0 4
[junit4:junit4] 2> 246172 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[16 (1438243074131099649)]} 0 4
[junit4:junit4] 2> 246188 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[17 (1438243074151022592)]} 0 1
[junit4:junit4] 2> 246188 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10017 (1438243074151022593)]} 0 1
[junit4:junit4] 2> 246193 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1438243074154168320)]} 0 1
[junit4:junit4] 2> 246193 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[17 (1438243074154168320)]} 0 3
[junit4:junit4] 2> 246193 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1438243074154168321)]} 0 1
[junit4:junit4] 2> 246194 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10017 (1438243074154168321)]} 0 4
[junit4:junit4] 2> 246204 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[12 (-1438243074168848384)]} 0 0
[junit4:junit4] 2> 246205 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10012 (-1438243074169896960)]} 0 0
[junit4:junit4] 2> 246208 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[12 (-1438243074170945536)]} 0 0
[junit4:junit4] 2> 246209 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10012 (-1438243074171994112)]} 0 0
[junit4:junit4] 2> 246209 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[12 (-1438243074170945536)]} 0 3
[junit4:junit4] 2> 246209 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10012 (-1438243074171994112)]} 0 2
[junit4:junit4] 2> 246212 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[18 (1438243074176188416)]} 0 1
[junit4:junit4] 2> 246212 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10018 (1438243074176188417)]} 0 1
[junit4:junit4] 2> 246217 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1438243074179334144)]} 0 0
[junit4:junit4] 2> 246218 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[18 (1438243074179334144)]} 0 4
[junit4:junit4] 2> 246218 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1438243074180382720)]} 0 1
[junit4:junit4] 2> 246219 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10018 (1438243074180382720)]} 0 5
[junit4:junit4] 2> 246243 T1022 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 246243 T1022 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:41032_si_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 246288 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[13 (-1438243074256928768)]} 0 0
[junit4:junit4] 2> 246289 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10013 (-1438243074257977344)]} 0 1
[junit4:junit4] 2> 246293 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10013 (-1438243074260074497)]} 0 0
[junit4:junit4] 2> 246293 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[13 (-1438243074260074496)]} 0 0
[junit4:junit4] 2> 246293 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10013 (-1438243074260074497)]} 0 2
[junit4:junit4] 2> 246294 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[13 (-1438243074260074496)]} 0 3
[junit4:junit4] 2> 246297 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10019 (1438243074265317377)]} 0 1
[junit4:junit4] 2> 246297 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[19 (1438243074265317376)]} 0 1
[junit4:junit4] 2> 246302 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1438243074268463104)]} 0 1
[junit4:junit4] 2> 246303 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10019 (1438243074268463104)]} 0 4
[junit4:junit4] 2> 246303 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1438243074268463105)]} 0 1
[junit4:junit4] 2> 246303 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[19 (1438243074268463105)]} 0 4
[junit4:junit4] 2> 246378 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10017 (-1438243074351300608)]} 0 0
[junit4:junit4] 2> 246378 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[17 (-1438243074351300609)]} 0 0
[junit4:junit4] 2> 246382 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10017 (-1438243074354446336)]} 0 0
[junit4:junit4] 2> 246383 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[17 (-1438243074354446337)]} 0 1
[junit4:junit4] 2> 246383 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10017 (-1438243074354446336)]} 0 3
[junit4:junit4] 2> 246383 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[17 (-1438243074354446337)]} 0 3
[junit4:junit4] 2> 246386 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10020 (1438243074358640640)]} 0 1
[junit4:junit4] 2> 246386 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[20 (1438243074358640641)]} 0 1
[junit4:junit4] 2> 246391 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1438243074361786368)]} 0 1
[junit4:junit4] 2> 246391 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10020 (1438243074361786368)]} 0 3
[junit4:junit4] 2> 246391 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1438243074361786369)]} 0 1
[junit4:junit4] 2> 246392 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[20 (1438243074361786369)]} 0 4
[junit4:junit4] 2> 246441 T984 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 246441 T984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 246444 T984 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=3core=collection1
[junit4:junit4] 2> 246449 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10018 (-1438243074425749504)]} 0 1
[junit4:junit4] 2> 246449 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[18 (-1438243074425749505)]} 0 0
[junit4:junit4] 2> 246453 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1438243074427846656)]} 0 0
[junit4:junit4] 2> 246453 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1438243074427846657)]} 0 0
[junit4:junit4] 2> 246454 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10018 (-1438243074427846656)]} 0 3
[junit4:junit4] 2> 246454 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[18 (-1438243074427846657)]} 0 3
[junit4:junit4] 2> 246457 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[21 (1438243074433089537)]} 0 1
[junit4:junit4] 2> 246457 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10021 (1438243074433089536)]} 0 1
[junit4:junit4] 2> 246462 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1438243074436235264)]} 0 1
[junit4:junit4] 2> 246462 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[21 (1438243074436235264)]} 0 3
[junit4:junit4] 2> 246462 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1438243074436235265)]} 0 1
[junit4:junit4] 2> 246463 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10021 (1438243074436235265)]} 0 4
[junit4:junit4] 2> 246546 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[19 (-1438243074527461376)]} 0 0
[junit4:junit4] 2> 246546 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10019 (-1438243074527461377)]} 0 0
[junit4:junit4] 2> 246550 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10019 (-1438243074529558529)]} 0 0
[junit4:junit4] 2> 246551 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[19 (-1438243074529558528)]} 0 1
[junit4:junit4] 2> 246551 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10019 (-1438243074529558529)]} 0 3
[junit4:junit4] 2> 246551 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[19 (-1438243074529558528)]} 0 3
[junit4:junit4] 2> 246554 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10022 (1438243074534801408)]} 0 1
[junit4:junit4] 2> 246554 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[22 (1438243074534801409)]} 0 1
[junit4:junit4] 2> 246559 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1438243074537947136)]} 0 1
[junit4:junit4] 2> 246560 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10022 (1438243074537947136)]} 0 4
[junit4:junit4] 2> 246560 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1438243074537947137)]} 0 1
[junit4:junit4] 2> 246560 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[22 (1438243074537947137)]} 0 4
[junit4:junit4] 2> 246605 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10022 (-1438243074589327360)]} 0 0
[junit4:junit4] 2> 246606 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[22 (-1438243074589327361)]} 0 1
[junit4:junit4] 2> 246609 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1438243074591424512)]} 0 0
[junit4:junit4] 2> 246609 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1438243074592473088)]} 0 0
[junit4:junit4] 2> 246610 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10022 (-1438243074591424512)]} 0 3
[junit4:junit4] 2> 246610 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[22 (-1438243074592473088)]} 0 3
[junit4:junit4] 2> 246613 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[23 (1438243074597715968)]} 0 1
[junit4:junit4] 2> 246613 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10023 (1438243074597715969)]} 0 1
[junit4:junit4] 2> 246619 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1438243074600861696)]} 0 1
[junit4:junit4] 2> 246619 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[23 (1438243074600861696)]} 0 3
[junit4:junit4] 2> 246619 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1438243074600861697)]} 0 1
[junit4:junit4] 2> 246620 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10023 (1438243074600861697)]} 0 4
[junit4:junit4] 2> 246708 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[23 (-1438243074697330688)]} 0 0
[junit4:junit4] 2> 246709 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10023 (-1438243074697330689)]} 0 1
[junit4:junit4] 2> 246713 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10023 (-1438243074700476416)]} 0 0
[junit4:junit4] 2> 246713 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[23 (-1438243074700476417)]} 0 0
[junit4:junit4] 2> 246714 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10023 (-1438243074700476416)]} 0 3
[junit4:junit4] 2> 246714 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[23 (-1438243074700476417)]} 0 3
[junit4:junit4] 2> 246717 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[24 (1438243074705719296)]} 0 1
[junit4:junit4] 2> 246717 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10024 (1438243074705719297)]} 0 1
[junit4:junit4] 2> 246722 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1438243074708865025)]} 0 1
[junit4:junit4] 2> 246723 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[24 (1438243074708865025)]} 0 4
[junit4:junit4] 2> 246723 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1438243074708865024)]} 0 1
[junit4:junit4] 2> 246724 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10024 (1438243074708865024)]} 0 5
[junit4:junit4] 2> 246821 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[25 (1438243074814771200)]} 0 1
[junit4:junit4] 2> 246821 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10025 (1438243074815819776)]} 0 0
[junit4:junit4] 2> 246827 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1438243074818965505)]} 0 1
[junit4:junit4] 2> 246827 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1438243074818965504)]} 0 1
[junit4:junit4] 2> 246827 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10025 (1438243074818965505)]} 0 4
[junit4:junit4] 2> 246827 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[25 (1438243074818965504)]} 0 4
[junit4:junit4] 2> 246921 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[26 (1438243074919628800)]} 0 1
[junit4:junit4] 2> 246921 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10026 (1438243074919628801)]} 0 1
[junit4:junit4] 2> 246926 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1438243074922774528)]} 0 1
[junit4:junit4] 2> 246926 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1438243074922774529)]} 0 1
[junit4:junit4] 2> 246927 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[26 (1438243074922774528)]} 0 4
[junit4:junit4] 2> 246927 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10026 (1438243074922774529)]} 0 4
[junit4:junit4] 2> 247017 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10027 (1438243075020292096)]} 0 1
[junit4:junit4] 2> 247017 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[27 (1438243075020292097)]} 0 1
[junit4:junit4] 2> 247023 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1438243075023437825)]} 0 1
[junit4:junit4] 2> 247023 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1438243075023437824)]} 0 1
[junit4:junit4] 2> 247023 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[27 (1438243075023437825)]} 0 4
[junit4:junit4] 2> 247023 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10027 (1438243075023437824)]} 0 4
[junit4:junit4] 2> 247042 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10026 (-1438243075047555072)]} 0 0
[junit4:junit4] 2> 247042 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[26 (-1438243075047555073)]} 0 0
[junit4:junit4] 2> 247046 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1438243075049652225)]} 0 0
[junit4:junit4] 2> 247046 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1438243075049652224)]} 0 0
[junit4:junit4] 2> 247047 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[26 (-1438243075049652225)]} 0 3
[junit4:junit4] 2> 247047 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10026 (-1438243075049652224)]} 0 3
[junit4:junit4] 2> 247050 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[28 (1438243075054895104)]} 0 1
[junit4:junit4] 2> 247050 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10028 (1438243075054895105)]} 0 1
[junit4:junit4] 2> 247055 T1038 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1438243075058040832)]} 0 1
[junit4:junit4] 2> 247055 T1035 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1438243075058040833)]} 0 1
[junit4:junit4] 2> 247055 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[28 (1438243075058040832)]} 0 3
[junit4:junit4] 2> 247056 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10028 (1438243075058040833)]} 0 4
[junit4:junit4] 2> 247140 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10029 (1438243075149266944)]} 0 1
[junit4:junit4] 2> 247140 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[29 (1438243075149266945)]} 0 1
[junit4:junit4] 2> 247145 T1037 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1438243075152412672)]} 0 1
[junit4:junit4] 2> 247146 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10029 (1438243075152412672)]} 0 4
[junit4:junit4] 2> 247146 T1036 C1429 P41032 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={distrib.from=http://127.0.0.1:57946/si_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1438243075152412673)]} 0 1
[junit4:junit4] 2> 247147 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[29 (1438243075152412673)]} 0 5
[junit4:junit4] 2> 247147 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 247148 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41032_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41032/si_"}
[junit4:junit4] 2> 247152 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 247152 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 247152 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 247152 T1042 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 247249 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10030 (1438243075263561728)]} 0 2
[junit4:junit4] 2> 247249 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[30 (1438243075263561729)]} 0 1
[junit4:junit4] 2> 247252 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10030 (1438243075266707456)]} 0 1
[junit4:junit4] 2> 247252 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[30 (1438243075267756032)]} 0 1
[junit4:junit4] 2> 247333 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10027 (-1438243075351642112)]} 0 0
[junit4:junit4] 2> 247333 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[27 (-1438243075352690688)]} 0 1
[junit4:junit4] 2> 247335 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[27 (-1438243075354787841)]} 0 0
[junit4:junit4] 2> 247335 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10027 (-1438243075354787840)]} 0 0
[junit4:junit4] 2> 247339 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[31 (1438243075357933568)]} 0 1
[junit4:junit4] 2> 247339 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10031 (1438243075357933569)]} 0 1
[junit4:junit4] 2> 247342 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[31 (1438243075361079296)]} 0 1
[junit4:junit4] 2> 247342 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10031 (1438243075361079297)]} 0 1
[junit4:junit4] 2> 247346 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10032 (1438243075366322176)]} 0 1
[junit4:junit4] 2> 247346 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[32 (1438243075365273600)]} 0 1
[junit4:junit4] 2> 247350 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[32 (1438243075369467904)]} 0 1
[junit4:junit4] 2> 247350 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10032 (1438243075369467905)]} 0 1
[junit4:junit4] 2> 247402 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10033 (1438243075423993856)]} 0 1
[junit4:junit4] 2> 247403 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[33 (1438243075423993857)]} 0 2
[junit4:junit4] 2> 247405 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10033 (1438243075428188160)]} 0 0
[junit4:junit4] 2> 247406 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[33 (1438243075428188161)]} 0 1
[junit4:junit4] 2> 247481 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[34 (1438243075506831361)]} 0 1
[junit4:junit4] 2> 247481 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10034 (1438243075506831360)]} 0 1
[junit4:junit4] 2> 247485 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10034 (1438243075511025664)]} 0 1
[junit4:junit4] 2> 247485 T1022 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[34 (1438243075511025665)]} 0 1
[junit4:junit4] 2> 247535 T999 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10028 (-1438243075563454464)]} 0 1
[junit4:junit4] 2> 247535 T1002 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[28 (-1438243075564503040)]} 0 1
[junit4:junit4] 2> 247537 T1020 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10028 (-1438243075566600192)]} 0 0
[junit4:junit4] 2> 247537 T1021 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[28 (-1438243075566600193)]} 0 0
[junit4:junit4] 2> 247541 T1001 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[35 (1438243075569745921)]} 0 1
[junit4:junit4] 2> 247541 T1000 C1427 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10035 (1438243075569745920)]} 0 1
[junit4:junit4] 2> 247544 T1019 C1428 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[35 (1438243075573940224)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C1430 name=collection1 org.apache.solr.core.SolrCore@4663c5fd url=http://127.0.0.1:57946/si_/collection1 node=127.0.0.1:57946_si_ C1430_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57946_si_, base_url=http://127.0.0.1:57946/si_, leader=true}
[junit4:junit4] 2> 247552 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10035 (1438243075573940225)]} 0 9
[junit4:junit4] 2> ASYNC NEW_CORE C1431 name=collection1 org.apache.solr.core.SolrCore@20be20df url=http://127.0.0.1:33035/si_/collection1 node=127.0.0.1:33035_si_ C1431_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:33035_si_, base_url=http://127.0.0.1:33035/si_, leader=true}
[junit4:junit4] 2> 247575 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[36 (1438243075605397504)]} 0 1
[junit4:junit4] 2> 247575 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10036 (1438243075606446080)]} 0 0
[junit4:junit4] 2> 247578 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[36 (1438243075609591808)]} 0 1
[junit4:junit4] 2> 247579 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10036 (1438243075609591809)]} 0 2
[junit4:junit4] 2> 247633 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10037 (1438243075666214912)]} 0 1
[junit4:junit4] 2> 247633 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[37 (1438243075666214913)]} 0 1
[junit4:junit4] 2> 247636 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10037 (1438243075669360640)]} 0 1
[junit4:junit4] 2> 247636 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[37 (1438243075669360641)]} 0 1
[junit4:junit4] 2> 247649 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10038 (1438243075682992128)]} 0 1
[junit4:junit4] 2> 247649 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[38 (1438243075682992129)]} 0 1
[junit4:junit4] 2> 247652 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10038 (1438243075686137856)]} 0 1
[junit4:junit4] 2> 247653 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[38 (1438243075686137857)]} 0 2
[junit4:junit4] 2> 247696 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[39 (1438243075732275201)]} 0 1
[junit4:junit4] 2> 247696 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10039 (1438243075732275200)]} 0 2
[junit4:junit4] 2> 247699 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10039 (1438243075735420928)]} 0 1
[junit4:junit4] 2> 247699 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[39 (1438243075735420929)]} 0 1
[junit4:junit4] 2> 247736 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[30 (-1438243075775266817)]} 0 1
[junit4:junit4] 2> 247736 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10030 (-1438243075775266816)]} 0 1
[junit4:junit4] 2> 247738 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[30 (-1438243075777363968)]} 0 0
[junit4:junit4] 2> 247738 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10030 (-1438243075777363969)]} 0 0
[junit4:junit4] 2> 247741 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[40 (1438243075780509696)]} 0 1
[junit4:junit4] 2> 247742 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10040 (1438243075780509697)]} 0 1
[junit4:junit4] 2> 247744 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[40 (1438243075783655424)]} 0 0
[junit4:junit4] 2> 247745 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10040 (1438243075783655425)]} 0 1
[junit4:junit4] 2> 247836 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10034 (-1438243075880124417)]} 0 0
[junit4:junit4] 2> 247836 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[34 (-1438243075880124416)]} 0 0
[junit4:junit4] 2> 247839 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[34 (-1438243075883270145)]} 0 0
[junit4:junit4] 2> 247839 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10034 (-1438243075883270144)]} 0 0
[junit4:junit4] 2> 247842 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[41 (1438243075885367296)]} 0 1
[junit4:junit4] 2> 247842 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10041 (1438243075885367297)]} 0 1
[junit4:junit4] 2> 247845 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[41 (1438243075888513024)]} 0 1
[junit4:junit4] 2> 247845 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10041 (1438243075888513025)]} 0 1
[junit4:junit4] 2> 247857 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[42 (1438243075901095936)]} 0 1
[junit4:junit4] 2> 247857 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10042 (1438243075901095937)]} 0 1
[junit4:junit4] 2> 247860 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[42 (1438243075904241664)]} 0 1
[junit4:junit4] 2> 247860 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10042 (1438243075905290240)]} 0 0
[junit4:junit4] 2> 247944 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10043 (1438243075992322048)]} 0 1
[junit4:junit4] 2> 247944 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[43 (1438243075992322049)]} 0 1
[junit4:junit4] 2> 247947 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10043 (1438243075996516352)]} 0 1
[junit4:junit4] 2> 247948 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[43 (1438243075996516353)]} 0 1
[junit4:junit4] 2> 248043 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10036 (-1438243076097179648)]} 0 0
[junit4:junit4] 2> 248043 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[36 (-1438243076097179649)]} 0 0
[junit4:junit4] 2> 248045 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10036 (-1438243076099276800)]} 0 0
[junit4:junit4] 2> 248046 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[36 (-1438243076100325376)]} 0 1
[junit4:junit4] 2> 248048 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10044 (1438243076102422528)]} 0 0
[junit4:junit4] 2> 248048 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[44 (1438243076102422529)]} 0 0
[junit4:junit4] 2> 248051 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10044 (1438243076104519680)]} 0 1
[junit4:junit4] 2> 248051 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[44 (1438243076105568256)]} 0 1
[junit4:junit4] 2> 248057 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[39 (-1438243076111859712)]} 0 0
[junit4:junit4] 2> 248057 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10039 (-1438243076111859713)]} 0 0
[junit4:junit4] 2> 248060 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[39 (-1438243076115005440)]} 0 1
[junit4:junit4] 2> 248060 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10039 (-1438243076115005441)]} 0 0
[junit4:junit4] 2> 248063 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[45 (1438243076117102593)]} 0 1
[junit4:junit4] 2> 248063 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10045 (1438243076117102592)]} 0 1
[junit4:junit4] 2> 248065 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[45 (1438243076120248320)]} 0 0
[junit4:junit4] 2> 248066 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10045 (1438243076120248321)]} 0 1
[junit4:junit4] 2> 248128 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[40 (-1438243076186308608)]} 0 0
[junit4:junit4] 2> 248128 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10040 (-1438243076186308609)]} 0 0
[junit4:junit4] 2> 248130 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[40 (-1438243076188405760)]} 0 0
[junit4:junit4] 2> 248131 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10040 (-1438243076189454336)]} 0 1
[junit4:junit4] 2> 248134 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[46 (1438243076191551488)]} 0 2
[junit4:junit4] 2> 248134 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10046 (1438243076191551489)]} 0 1
[junit4:junit4] 2> 248136 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[46 (1438243076194697216)]} 0 0
[junit4:junit4] 2> 248137 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10046 (1438243076194697217)]} 0 1
[junit4:junit4] 2> 248175 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[41 (-1438243076235591680)]} 0 0
[junit4:junit4] 2> 248175 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10041 (-1438243076235591681)]} 0 0
[junit4:junit4] 2> 248177 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[41 (-1438243076237688832)]} 0 0
[junit4:junit4] 2> 248178 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10041 (-1438243076238737408)]} 0 1
[junit4:junit4] 2> 248181 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10047 (1438243076240834561)]} 0 1
[junit4:junit4] 2> 248181 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[47 (1438243076240834560)]} 0 1
[junit4:junit4] 2> 248184 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10047 (1438243076243980288)]} 0 1
[junit4:junit4] 2> 248184 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[47 (1438243076243980289)]} 0 1
[junit4:junit4] 2> 248211 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10048 (1438243076273340416)]} 0 1
[junit4:junit4] 2> 248212 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[48 (1438243076273340417)]} 0 1
[junit4:junit4] 2> 248214 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10048 (1438243076276486144)]} 0 1
[junit4:junit4] 2> 248215 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[48 (1438243076276486145)]} 0 1
[junit4:junit4] 2> 248220 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10049 (1438243076282777600)]} 0 1
[junit4:junit4] 2> 248220 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[49 (1438243076282777601)]} 0 0
[junit4:junit4] 2> 248223 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10049 (1438243076285923328)]} 0 1
[junit4:junit4] 2> 248224 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[49 (1438243076285923329)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C1432 name=collection1 org.apache.solr.core.SolrCore@4baeebfd url=http://127.0.0.1:41032/si_/collection1 node=127.0.0.1:41032_si_ C1432_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41032_si_, base_url=http://127.0.0.1:41032/si_}
[junit4:junit4] 2> 248244 T1045 C1432 P41032 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:57946/si_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 248244 T1045 C1432 P41032 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41032/si_ START replicas=[http://127.0.0.1:57946/si_/collection1/] nUpdates=100
[junit4:junit4] 2> 248246 T1045 C1432 P41032 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 248246 T1045 C1432 P41032 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 248246 T1045 C1432 P41032 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 248246 T1045 C1432 P41032 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 248246 T1019 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 248247 T1045 C1432 P41032 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 248247 T1045 C1432 P41032 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:57946/si_/collection1/. core=collection1
[junit4:junit4] 2> 248247 T1045 C1432 P41032 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 248250 T1020 C1430 P57946 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 248277 T1020 C1430 P57946 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@53e3b7f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a6e08a4)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@53e3b7f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a6e08a4)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 248278 T1020 C1430 P57946 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 248279 T1020 C1430 P57946 oass.SolrIndexSearcher.<init> Opening Searcher@4d22c336 realtime
[junit4:junit4] 2> 248279 T1020 C1430 P57946 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 248279 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4] 2> 248280 T1045 C1432 P41032 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 248280 T1045 C1432 P41032 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 248282 T1021 C1430 P57946 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 248283 T1021 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 248283 T1045 C1432 P41032 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 248283 T1045 C1432 P41032 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 248283 T1045 C1432 P41032 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 248285 T1019 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 248286 T1045 C1432 P41032 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 248286 T1045 C1432 P41032 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index.20130619001802666
[junit4:junit4] 2> 248287 T1045 C1432 P41032 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@3697e8c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6484439e) fullCopy=false
[junit4:junit4] 2> 248289 T1020 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 248291 T1022 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 248294 T1021 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 248297 T1019 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 248300 T1020 C1430 P57946 oasc.SolrCore.execute [collection1] webapp=/si_ path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 248300 T1045 C1432 P41032 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 248302 T1045 C1432 P41032 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 248302 T1045 C1432 P41032 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 248302 T1045 C1432 P41032 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 248318 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10050 (1438243076385538049)]} 0 0
[junit4:junit4] 2> 248319 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[50 (1438243076385538048)]} 0 1
[junit4:junit4] 2> 248322 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10050 (1438243076388683776)]} 0 1
[junit4:junit4] 2> 248322 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[50 (1438243076388683777)]} 0 1
[junit4:junit4] 2> 248326 T1045 C1432 P41032 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 248327 T1045 C1432 P41032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 248329 T1045 C1432 P41032 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 248329 T1045 C1432 P41032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 248329 T1045 C1432 P41032 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 248331 T1045 C1432 P41032 oass.SolrIndexSearcher.<init> Opening Searcher@2d3b749e main
[junit4:junit4] 2> 248331 T1044 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d3b749e main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c46/22)}
[junit4:junit4] 2> 248332 T1045 C1432 P41032 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index.20130619001802666 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index.20130619001802666;done=true>>]
[junit4:junit4] 2> 248332 T1045 C1432 P41032 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index.20130619001802666
[junit4:junit4] 2> 248332 T1045 C1432 P41032 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index.20130619001802666
[junit4:junit4] 2> 248335 T1045 C1432 P41032 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 248335 T1053 C1432 P41032 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=6717
[junit4:junit4] 2> 248337 T1053 C1432 P41032 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 248337 T1053 C1432 P41032 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a1e20d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1eab86cb)),segFN=segments_3,generation=3}
[junit4:junit4] 2> 248338 T1053 C1432 P41032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 248338 T1053 C1432 P41032 oass.SolrIndexSearcher.<init> Opening Searcher@4a6af5e9 main
[junit4:junit4] 2> 248338 T1053 C1432 P41032 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 248339 T1044 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a6af5e9 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c46/22)}
[junit4:junit4] 2> 248339 T1053 C1432 P41032 oasup.LogUpdateProcessor.finish [collection1] {} 0 4
[junit4:junit4] 2> 248339 T1053 C1432 P41032 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=6717}
[junit4:junit4] 2> 248340 T1045 C1432 P41032 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 248340 T1045 C1432 P41032 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 248340 T1045 C1432 P41032 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 248342 T1045 C1432 P41032 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 248343 T984 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4baeebfd
[junit4:junit4] 2> 248347 T984 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=46,cumulative_deletesById=24,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 248348 T984 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 248348 T984 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 248349 T984 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 248350 T984 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 248350 T984 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 248350 T984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index;done=false>>]
[junit4:junit4] 2> 248351 T984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index
[junit4:junit4] 2> 248351 T984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2;done=false>>]
[junit4:junit4] 2> 248351 T984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2
[junit4:junit4] 2> 248352 T1042 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 248353 T1042 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 248353 T1042 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 248352 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 248353 T1013 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 248354 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 248354 T1006 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 248354 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 248355 T1026 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 248374 T984 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/si_,null}
[junit4:junit4] 2> 248407 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10051 (1438243076477812737)]} 0 1
[junit4:junit4] 2> 248407 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[51 (1438243076477812736)]} 0 1
[junit4:junit4] 2> 248410 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[51 (1438243076482007041)]} 0 0
[junit4:junit4] 2> 248411 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10051 (1438243076482007040)]} 0 1
[junit4:junit4] 2> 248474 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10042 (-1438243076548067329)]} 0 1
[junit4:junit4] 2> 248474 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[42 (-1438243076548067328)]} 0 1
[junit4:junit4] 2> 248477 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10042 (-1438243076552261633)]} 0 0
[junit4:junit4] 2> 248477 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[42 (-1438243076552261632)]} 0 0
[junit4:junit4] 2> 248480 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10052 (1438243076555407360)]} 0 1
[junit4:junit4] 2> 248480 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[52 (1438243076555407361)]} 0 1
[junit4:junit4] 2> 248484 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[52 (1438243076558553089)]} 0 1
[junit4:junit4] 2> 248484 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10052 (1438243076558553088)]} 0 1
[junit4:junit4] 2> 248553 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10053 (1438243076630904832)]} 0 1
[junit4:junit4] 2> 248553 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[53 (1438243076630904833)]} 0 1
[junit4:junit4] 2> 248556 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10053 (1438243076634050560)]} 0 1
[junit4:junit4] 2> 248556 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[53 (1438243076635099136)]} 0 1
[junit4:junit4] 2> 248621 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10054 (1438243076703256576)]} 0 1
[junit4:junit4] 2> 248621 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[54 (1438243076703256577)]} 0 1
[junit4:junit4] 2> 248624 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[54 (1438243076706402305)]} 0 0
[junit4:junit4] 2> 248625 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10054 (1438243076706402304)]} 0 2
[junit4:junit4] 2> 248632 T999 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[55 (1438243076714790912)]} 0 0
[junit4:junit4] 2> 248633 T1002 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10055 (1438243076714790913)]} 0 1
[junit4:junit4] 2> 248636 T1021 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[55 (1438243076717936640)]} 0 1
[junit4:junit4] 2> 248636 T1022 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10055 (1438243076717936641)]} 0 1
[junit4:junit4] 2> 248656 T1007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 248657 T1007 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41032_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41032/si_"}
[junit4:junit4] 2> 248661 T1026 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 248661 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 248661 T1013 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 248664 T1001 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[43 (-1438243076748345344)]} 0 0
[junit4:junit4] 2> 248664 T1000 C1431 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10043 (-1438243076748345345)]} 0 0
[junit4:junit4] 2> 248667 T1020 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[43 (-1438243076751491072)]} 0 0
[junit4:junit4] 2> 248667 T1019 C1430 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10043 (-1438243076751491073)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C1433 name=collection1 org.apache.solr.core.SolrCore@20be20df url=http://127.0.0.1:33035/si_/collection1 node=127.0.0.1:33035_si_ C1433_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:33035_si_, base_url=http://127.0.0.1:33035/si_, leader=true}
[junit4:junit4] 2> 248680 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[56 (1438243076753588224)]} 0 11
[junit4:junit4] 2> 248681 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10056 (1438243076754636800)]} 0 11
[junit4:junit4] 2> ASYNC NEW_CORE C1434 name=collection1 org.apache.solr.core.SolrCore@4663c5fd url=http://127.0.0.1:57946/si_/collection1 node=127.0.0.1:57946_si_ C1434_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57946_si_, base_url=http://127.0.0.1:57946/si_, leader=true}
[junit4:junit4] 2> 248683 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[56 (1438243076768268288)]} 0 0
[junit4:junit4] 2> 248684 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10056 (1438243076768268289)]} 0 1
[junit4:junit4] 2> 248757 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[45 (-1438243076845862913)]} 0 0
[junit4:junit4] 2> 248757 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10045 (-1438243076845862912)]} 0 0
[junit4:junit4] 2> 248760 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[45 (-1438243076849008640)]} 0 1
[junit4:junit4] 2> 248760 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10045 (-1438243076849008641)]} 0 1
[junit4:junit4] 2> 248762 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[57 (1438243076851105792)]} 0 0
[junit4:junit4] 2> 248763 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10057 (1438243076851105793)]} 0 1
[junit4:junit4] 2> 248765 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[57 (1438243076853202944)]} 0 1
[junit4:junit4] 2> 248765 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10057 (1438243076854251520)]} 0 0
[junit4:junit4] 2> 248799 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10046 (-1438243076888854529)]} 0 1
[junit4:junit4] 2> 248799 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[46 (-1438243076888854528)]} 0 1
[junit4:junit4] 2> 248801 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10046 (-1438243076892000256)]} 0 0
[junit4:junit4] 2> 248801 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[46 (-1438243076892000257)]} 0 0
[junit4:junit4] 2> 248803 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10058 (1438243076894097408)]} 0 0
[junit4:junit4] 2> 248804 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[58 (1438243076894097409)]} 0 1
[junit4:junit4] 2> 248806 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10058 (1438243076897243136)]} 0 1
[junit4:junit4] 2> 248806 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[58 (1438243076897243137)]} 0 0
[junit4:junit4] 2> 248868 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[59 (1438243076962254849)]} 0 0
[junit4:junit4] 2> 248869 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10059 (1438243076962254848)]} 0 1
[junit4:junit4] 2> 248871 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10059 (1438243076965400577)]} 0 0
[junit4:junit4] 2> 248872 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[59 (1438243076965400576)]} 0 1
[junit4:junit4] 2> 248904 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[47 (-1438243077000003585)]} 0 0
[junit4:junit4] 2> 248904 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10047 (-1438243077000003584)]} 0 0
[junit4:junit4] 2> 248907 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[47 (-1438243077002100736)]} 0 1
[junit4:junit4] 2> 248907 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10047 (-1438243077003149312)]} 0 0
[junit4:junit4] 2> 248909 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[60 (1438243077005246464)]} 0 0
[junit4:junit4] 2> 248910 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10060 (1438243077005246465)]} 0 1
[junit4:junit4] 2> 248912 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[60 (1438243077008392192)]} 0 0
[junit4:junit4] 2> 248913 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10060 (1438243077008392193)]} 0 1
[junit4:junit4] 2> 248946 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[48 (-1438243077044043776)]} 0 0
[junit4:junit4] 2> 248946 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10048 (-1438243077044043777)]} 0 0
[junit4:junit4] 2> 248949 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[48 (-1438243077047189504)]} 0 1
[junit4:junit4] 2> 248949 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10048 (-1438243077047189505)]} 0 0
[junit4:junit4] 2> 248952 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[61 (1438243077049286656)]} 0 1
[junit4:junit4] 2> 248952 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10061 (1438243077049286657)]} 0 1
[junit4:junit4] 2> 248954 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[61 (1438243077052432384)]} 0 0
[junit4:junit4] 2> 248955 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10061 (1438243077052432385)]} 0 1
[junit4:junit4] 2> 249017 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[50 (-1438243077118492672)]} 0 0
[junit4:junit4] 2> 249017 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10050 (-1438243077118492673)]} 0 0
[junit4:junit4] 2> 249020 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[50 (-1438243077121638400)]} 0 1
[junit4:junit4] 2> 249020 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10050 (-1438243077121638401)]} 0 0
[junit4:junit4] 2> 249022 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[62 (1438243077123735552)]} 0 0
[junit4:junit4] 2> 249023 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10062 (1438243077123735553)]} 0 1
[junit4:junit4] 2> 249025 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[62 (1438243077126881280)]} 0 0
[junit4:junit4] 2> 249026 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10062 (1438243077126881281)]} 0 1
[junit4:junit4] 2> 249094 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[51 (-1438243077199233024)]} 0 1
[junit4:junit4] 2> 249094 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10051 (-1438243077199233025)]} 0 0
[junit4:junit4] 2> 249096 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[51 (-1438243077201330176)]} 0 0
[junit4:junit4] 2> 249096 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10051 (-1438243077201330177)]} 0 0
[junit4:junit4] 2> 249099 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[63 (1438243077203427328)]} 0 1
[junit4:junit4] 2> 249099 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10063 (1438243077203427329)]} 0 1
[junit4:junit4] 2> 249102 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[63 (1438243077206573056)]} 0 1
[junit4:junit4] 2> 249102 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10063 (1438243077206573057)]} 0 1
[junit4:junit4] 2> 249124 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10056 (-1438243077230690304)]} 0 0
[junit4:junit4] 2> 249124 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[56 (-1438243077230690305)]} 0 0
[junit4:junit4] 2> 249127 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10056 (-1438243077233836032)]} 0 1
[junit4:junit4] 2> 249127 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[56 (-1438243077233836033)]} 0 0
[junit4:junit4] 2> 249129 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10064 (1438243077235933184)]} 0 1
[junit4:junit4] 2> 249130 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[64 (1438243077235933185)]} 0 1
[junit4:junit4] 2> 249132 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10064 (1438243077238030336)]} 0 1
[junit4:junit4] 2> 249132 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[64 (1438243077239078912)]} 0 0
[junit4:junit4] 2> 249162 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10057 (-1438243077270536192)]} 0 0
[junit4:junit4] 2> 249162 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[57 (-1438243077270536193)]} 0 0
[junit4:junit4] 2> 249164 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10057 (-1438243077272633344)]} 0 0
[junit4:junit4] 2> 249165 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[57 (-1438243077272633345)]} 0 1
[junit4:junit4] 2> 249167 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10065 (1438243077274730496)]} 0 1
[junit4:junit4] 2> 249167 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[65 (1438243077275779072)]} 0 1
[junit4:junit4] 2> 249170 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10065 (1438243077277876224)]} 0 1
[junit4:junit4] 2> 249170 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[65 (1438243077277876225)]} 0 1
[junit4:junit4] 2> 249216 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10066 (1438243077327159296)]} 0 1
[junit4:junit4] 2> 249216 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[66 (1438243077327159297)]} 0 1
[junit4:junit4] 2> 249219 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10066 (1438243077330305024)]} 0 0
[junit4:junit4] 2> 249220 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[66 (1438243077330305025)]} 0 1
[junit4:junit4] 2> 249264 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10059 (-1438243077377490944)]} 0 0
[junit4:junit4] 2> 249264 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[59 (-1438243077377490945)]} 0 0
[junit4:junit4] 2> 249267 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10059 (-1438243077380636672)]} 0 1
[junit4:junit4] 2> 249267 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[59 (-1438243077380636673)]} 0 0
[junit4:junit4] 2> 249270 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10067 (1438243077382733824)]} 0 0
[junit4:junit4] 2> 249270 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[67 (1438243077382733825)]} 0 1
[junit4:junit4] 2> 249272 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10067 (1438243077385879552)]} 0 0
[junit4:junit4] 2> 249273 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[67 (1438243077385879553)]} 0 1
[junit4:junit4] 2> 249343 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10063 (-1438243077460328448)]} 0 0
[junit4:junit4] 2> 249343 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[63 (-1438243077460328449)]} 0 0
[junit4:junit4] 2> 249346 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10063 (-1438243077463474176)]} 0 1
[junit4:junit4] 2> 249346 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[63 (-1438243077463474177)]} 0 0
[junit4:junit4] 2> 249348 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10068 (1438243077465571328)]} 0 0
[junit4:junit4] 2> 249349 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[68 (1438243077465571329)]} 0 1
[junit4:junit4] 2> 249352 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10068 (1438243077468717056)]} 0 1
[junit4:junit4] 2> 249352 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[68 (1438243077468717057)]} 0 1
[junit4:junit4] 2> 249368 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10064 (-1438243077486542848)]} 0 0
[junit4:junit4] 2> 249368 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[64 (-1438243077486542849)]} 0 0
[junit4:junit4] 2> 249371 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10064 (-1438243077489688576)]} 0 0
[junit4:junit4] 2> 249371 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[64 (-1438243077489688577)]} 0 0
[junit4:junit4] 2> 249373 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10069 (1438243077491785728)]} 0 0
[junit4:junit4] 2> 249374 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[69 (1438243077491785729)]} 0 1
[junit4:junit4] 2> 249376 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10069 (1438243077494931456)]} 0 1
[junit4:junit4] 2> 249377 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[69 (1438243077494931457)]} 0 1
[junit4:junit4] 2> 249478 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10066 (-1438243077601886208)]} 0 0
[junit4:junit4] 2> 249478 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[66 (-1438243077601886209)]} 0 0
[junit4:junit4] 2> 249481 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10066 (-1438243077603983360)]} 0 1
[junit4:junit4] 2> 249481 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[66 (-1438243077605031936)]} 0 1
[junit4:junit4] 2> 249483 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10070 (1438243077607129088)]} 0 1
[junit4:junit4] 2> 249484 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[70 (1438243077607129089)]} 0 1
[junit4:junit4] 2> 249486 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10070 (1438243077609226240)]} 0 1
[junit4:junit4] 2> 249486 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[70 (1438243077610274816)]} 0 0
[junit4:junit4] 2> 249534 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10071 (1438243077660606464)]} 0 0
[junit4:junit4] 2> 249534 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[71 (1438243077660606465)]} 0 0
[junit4:junit4] 2> 249537 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10071 (1438243077663752192)]} 0 0
[junit4:junit4] 2> 249538 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[71 (1438243077663752193)]} 0 1
[junit4:junit4] 2> 249615 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[68 (-1438243077745541120)]} 0 0
[junit4:junit4] 2> 249615 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10068 (-1438243077745541121)]} 0 0
[junit4:junit4] 2> 249617 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[68 (-1438243077747638272)]} 0 0
[junit4:junit4] 2> 249618 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10068 (-1438243077748686848)]} 0 1
[junit4:junit4] 2> 249620 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[72 (1438243077750784000)]} 0 1
[junit4:junit4] 2> 249620 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10072 (1438243077750784001)]} 0 0
[junit4:junit4] 2> 249623 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[72 (1438243077752881152)]} 0 1
[junit4:junit4] 2> 249623 T1020 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10072 (1438243077752881153)]} 0 1
[junit4:junit4] 2> 249654 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[73 (1438243077785387009)]} 0 1
[junit4:junit4] 2> 249654 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10073 (1438243077785387008)]} 0 1
[junit4:junit4] 2> 249657 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[73 (1438243077788532736)]} 0 1
[junit4:junit4] 2> 249658 T1021 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10073 (1438243077788532737)]} 0 2
[junit4:junit4] 2> 249719 T1002 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10069 (-1438243077854593024)]} 0 0
[junit4:junit4] 2> 249719 T1001 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[69 (-1438243077854593025)]} 0 0
[junit4:junit4] 2> 249722 T1022 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[10069 (-1438243077857738752)]} 0 0
[junit4:junit4] 2> 249723 T1019 C1434 P57946 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {delete=[69 (-1438243077858787328)]} 0 1
[junit4:junit4] 2> 249725 T1000 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[10074 (1438243077859835904)]} 0 1
[junit4:junit4] 2> 249725 T999 C1433 P33035 oasup.LogUpdateProcessor.finish [collection1] webapp=/si_ path=/update params={wt=javabin&version=2} {add=[74 (1438243077860884480)]} 0 0
[junit4:junit4] 2> 249728 T102
[...truncated too long message...]
ncToMe http://127.0.0.1:41032/si_/collection1/ has no replicas
[junit4:junit4] 2> 344774 T1066 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41032/si_/collection1/
[junit4:junit4] 2> 344775 T1066 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 344775 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 344776 T1078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57946_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57946/si_"}
[junit4:junit4] 2> 344779 T1078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57946_si_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57946/si_"}
[junit4:junit4] 2> 344785 T984 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/si_,null}
[junit4:junit4] 2> 344786 T1066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 344836 T984 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 41032
[junit4:junit4] 2> 344837 T984 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=80194999
[junit4:junit4] 2> 344978 T1006 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 344978 T1006 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 344978 T1006 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 345838 T984 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 345839 T984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 345840 T984 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7db8d5b4
[junit4:junit4] 2> 345844 T984 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=66,cumulative_deletesById=44,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 345845 T984 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 345845 T984 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 345846 T984 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 345847 T984 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 345847 T984 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 345847 T984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2;done=false>>]
[junit4:junit4] 2> 345847 T984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2
[junit4:junit4] 2> 345848 T984 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index;done=false>>]
[junit4:junit4] 2> 345848 T984 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371615470614/jetty2/index
[junit4:junit4] 2> 345849 T1078 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89890191482290186-127.0.0.1:41032_si_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 345849 T1066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 345850 T1066 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 345850 T1066 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 345870 T984 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/si_,null}
[junit4:junit4] 2> 345927 T984 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 345930 T984 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46157 46157
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=51B29FEFC84DEB49 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_FR -Dtests.timezone=Canada/Eastern -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 110s J0 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<184> but was:<32>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([51B29FEFC84DEB49:D05411F7BF128B75]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 345976 T984 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 109750 T983 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=Pulsing41(freqCutoff=9 minBlockSize=40 maxBlockSize=96), text=PostingsFormat(name=Memory doPackFST= false), _version_=MockVariableIntBlock(baseBlockSize=120), rnd_b=PostingsFormat(name=MockRandom), intDefault=MockVariableIntBlock(baseBlockSize=120), timestamp=Pulsing41(freqCutoff=9 minBlockSize=40 maxBlockSize=96), id=PostingsFormat(name=MockRandom), a_t=Pulsing41(freqCutoff=9 minBlockSize=40 maxBlockSize=96), range_facet_sl=PostingsFormat(name=MockRandom), range_facet_si=Pulsing41(freqCutoff=9 minBlockSize=40 maxBlockSize=96), other_tl1=MockVariableIntBlock(baseBlockSize=120), multiDefault=Pulsing41(freqCutoff=9 minBlockSize=40 maxBlockSize=96), a_si=MockVariableIntBlock(baseBlockSize=120)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=DefaultSimilarity, locale=fr_FR, timezone=Canada/Eastern
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=48409760,total=149942272
[junit4:junit4] 2> NOTE: All tests run in this JVM: [QueryEqualityTest, SolrInfoMBeanTest, TestPropInjectDefaults, SuggesterTest, LoggingHandlerTest, AlternateDirectoryTest, PolyFieldTest, TestSort, FileBasedSpellCheckerTest, CacheHeaderTest, TestFastLRUCache, TestStressVersions, TestFastWriter, StatelessScriptUpdateProcessorFactoryTest, PreAnalyzedFieldTest, QueryResultKeyTest, TestCharFilters, SystemInfoHandlerTest, DistributedSpellCheckComponentTest, TestClassNameShortening, PluginInfoTest, SyncSliceTest, BasicDistributedZkTest, DocValuesTest, PrimitiveFieldTypeTest, SolrIndexSplitterTest, CircularListTest, TestFunctionQuery, PreAnalyzedUpdateProcessorTest, DocumentBuilderTest, DistributedTermsComponentTest, TestValueSourceCache, RecoveryZkTest]
[junit4:junit4] Completed on J0 in 110.52s, 1 test, 1 failure <<< FAILURES!
[...truncated 708 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1252 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 40 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure