You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/06/15 06:38:49 UTC
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4067 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4067/
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 22 from http://127.0.0.1:15122/collection1lastClient and got 3 from http://127.0.0.1:15139/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 22 from http://127.0.0.1:15122/collection1lastClient and got 3 from http://127.0.0.1:15139/collection1
at __randomizedtesting.SeedInfo.seed([45EBF799E0CF3878:C40D798197905844]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<209> but was:<69>
Stack Trace:
java.lang.AssertionError: expected:<209> but was:<69>
at __randomizedtesting.SeedInfo.seed([45EBF799E0CF3878:C40D798197905844]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9401 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 322671 T836 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 322679 T836 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371269417899
[junit4:junit4] 2> 322681 T836 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 322682 T837 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 322784 T836 oasc.ZkTestServer.run start zk server on port:42111
[junit4:junit4] 2> 322785 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 322795 T843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1173d0c5 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 322795 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 322795 T836 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 322813 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 322820 T845 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@238d4b23 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 322820 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 322821 T836 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 322837 T836 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 322845 T836 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 322848 T836 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 322851 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 322852 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 322864 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 322864 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 322970 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 322971 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 322974 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 322975 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 322986 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 322987 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 322990 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 322991 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 322994 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 322995 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 322998 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 322999 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 323002 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 323002 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 323291 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 323296 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16507
[junit4:junit4] 2> 323297 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 323298 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 323298 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231
[junit4:junit4] 2> 323299 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/solr.xml
[junit4:junit4] 2> 323299 T836 oasc.CoreContainer.<init> New CoreContainer 959506201
[junit4:junit4] 2> 323300 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/'
[junit4:junit4] 2> 323300 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/'
[junit4:junit4] 2> 323435 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 323436 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 323436 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 323437 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 323437 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 323438 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 323438 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 323439 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 323439 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 323440 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 323460 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 323461 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 323462 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 323463 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 323466 T856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@430df842 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 323467 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 323484 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 323499 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 323501 T858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f94d995 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 323501 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 323508 T836 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 323512 T836 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 323521 T836 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 323523 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16507_
[junit4:junit4] 2> 323525 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16507_
[junit4:junit4] 2> 323535 T836 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 323547 T836 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 323554 T836 oasc.Overseer.start Overseer (id=89867512571953155-127.0.0.1:16507_-n_0000000000) starting
[junit4:junit4] 2> 323559 T836 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 323570 T860 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 323571 T836 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 323573 T836 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 323575 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 323577 T859 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 323580 T861 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 323580 T861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 325081 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 325082 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:16507_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:16507"}
[junit4:junit4] 2> 325083 T859 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 325083 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 325105 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 325584 T861 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1
[junit4:junit4] 2> 325584 T861 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 325585 T861 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 325585 T861 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 325587 T861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/'
[junit4:junit4] 2> 325590 T861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 325590 T861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/lib/README' to classloader
[junit4:junit4] 2> 325653 T861 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 325723 T861 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 325826 T861 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 325831 T861 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 326410 T861 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 326424 T861 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 326428 T861 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 326442 T861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 326447 T861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 326452 T861 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 326453 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 326453 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 326453 T861 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 326454 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 326455 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 326455 T861 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 326518 T861 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/
[junit4:junit4] 2> 326519 T861 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 326519 T861 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 326521 T861 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data
[junit4:junit4] 2> 326522 T861 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index/
[junit4:junit4] 2> 326523 T861 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 326524 T861 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index
[junit4:junit4] 2> 326547 T861 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 326548 T861 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 326550 T861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 326551 T861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 326551 T861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 326553 T861 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 326553 T861 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 326571 T861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 326585 T861 oass.SolrIndexSearcher.<init> Opening Searcher@4b8d9424 main
[junit4:junit4] 2> 326587 T861 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 326587 T861 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 326591 T862 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b8d9424 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 326593 T861 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 326593 T861 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16507 collection:control_collection shard:shard1
[junit4:junit4] 2> 326594 T861 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 326612 T861 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 326615 T861 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 326615 T861 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 326616 T861 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16507/collection1/
[junit4:junit4] 2> 326616 T861 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 326616 T861 oasc.SyncStrategy.syncToMe http://127.0.0.1:16507/collection1/ has no replicas
[junit4:junit4] 2> 326616 T861 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16507/collection1/
[junit4:junit4] 2> 326617 T861 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 328113 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 328130 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 328175 T861 oasc.ZkController.register We are http://127.0.0.1:16507/collection1/ and leader is http://127.0.0.1:16507/collection1/
[junit4:junit4] 2> 328175 T861 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16507
[junit4:junit4] 2> 328175 T861 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 328175 T861 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 328175 T861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 328177 T861 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 328178 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 328178 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 328179 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 328191 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 328192 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 328194 T865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@628441a7 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 328194 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 328196 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 328198 T836 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 328425 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 328427 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:53832
[junit4:junit4] 2> 328428 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 328428 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 328428 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416
[junit4:junit4] 2> 328429 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/solr.xml
[junit4:junit4] 2> 328429 T836 oasc.CoreContainer.<init> New CoreContainer 233765018
[junit4:junit4] 2> 328429 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/'
[junit4:junit4] 2> 328430 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/'
[junit4:junit4] 2> 328518 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 328519 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 328519 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 328519 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 328520 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 328520 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 328520 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 328521 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 328521 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 328521 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 328533 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 328534 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 328534 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 328535 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 328537 T876 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@780d5011 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 328538 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 328540 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 328548 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 328551 T878 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e91bdf8 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 328551 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 328564 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 329570 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53832_
[junit4:junit4] 2> 329576 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53832_
[junit4:junit4] 2> 329581 T865 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 329581 T878 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 329581 T858 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 329581 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 329595 T879 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 329596 T879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 329645 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 329666 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:16507__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:16507_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:16507"}
[junit4:junit4] 2> 329671 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53832_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53832"}
[junit4:junit4] 2> 329671 T859 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 329671 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 329691 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 329691 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 329691 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 330599 T879 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1
[junit4:junit4] 2> 330599 T879 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 330600 T879 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 330600 T879 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 330602 T879 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/'
[junit4:junit4] 2> 330603 T879 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 330603 T879 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/lib/README' to classloader
[junit4:junit4] 2> 330644 T879 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 330692 T879 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 330793 T879 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 330798 T879 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 331300 T879 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 331310 T879 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 331313 T879 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 331327 T879 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 331331 T879 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 331335 T879 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 331336 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 331337 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 331337 T879 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 331338 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 331338 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 331339 T879 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 331339 T879 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/
[junit4:junit4] 2> 331339 T879 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 331340 T879 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 331340 T879 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1
[junit4:junit4] 2> 331341 T879 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index/
[junit4:junit4] 2> 331342 T879 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 331343 T879 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index
[junit4:junit4] 2> 331347 T879 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 331348 T879 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 331350 T879 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 331350 T879 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 331351 T879 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 331351 T879 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 331352 T879 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 331352 T879 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 331352 T879 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 331353 T879 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 331353 T879 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 331367 T879 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 331374 T879 oass.SolrIndexSearcher.<init> Opening Searcher@4194c10f main
[junit4:junit4] 2> 331375 T879 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 331375 T879 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 331379 T880 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4194c10f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 331380 T879 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 331380 T879 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53832 collection:collection1 shard:shard1
[junit4:junit4] 2> 331381 T879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 331396 T879 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 331405 T879 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 331405 T879 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 331406 T879 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 331406 T879 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 331406 T879 oasc.SyncStrategy.syncToMe http://127.0.0.1:53832/collection1/ has no replicas
[junit4:junit4] 2> 331406 T879 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 331406 T879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 332700 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 332717 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 332717 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 332717 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 332755 T879 oasc.ZkController.register We are http://127.0.0.1:53832/collection1/ and leader is http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 332755 T879 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53832
[junit4:junit4] 2> 332755 T879 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 332755 T879 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 332755 T879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 332758 T879 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 332759 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 332759 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 332760 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 332951 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 332954 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15080
[junit4:junit4] 2> 332954 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 332955 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 332955 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989
[junit4:junit4] 2> 332955 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/solr.xml
[junit4:junit4] 2> 332956 T836 oasc.CoreContainer.<init> New CoreContainer 277165989
[junit4:junit4] 2> 332956 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/'
[junit4:junit4] 2> 332956 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/'
[junit4:junit4] 2> 333041 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 333042 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 333043 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 333043 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 333043 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 333044 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 333044 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 333044 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 333045 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 333045 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 333056 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 333057 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 333057 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 333058 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 333060 T892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a7b1e0 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 333061 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 333072 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 333085 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 333087 T894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36df849d name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 333087 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 333105 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 334110 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15080_
[junit4:junit4] 2> 334126 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15080_
[junit4:junit4] 2> 334209 T878 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T858 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T894 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T865 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334209 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334210 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334282 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 334283 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53832__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53832_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53832"}
[junit4:junit4] 2> 334285 T895 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 334285 T895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 334351 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15080_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15080"}
[junit4:junit4] 2> 334352 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 334352 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 334375 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334376 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334375 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334376 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 335331 T895 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1
[junit4:junit4] 2> 335331 T895 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 335332 T895 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 335332 T895 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 335333 T895 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/'
[junit4:junit4] 2> 335334 T895 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/lib/README' to classloader
[junit4:junit4] 2> 335335 T895 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 335370 T895 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 335411 T895 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 335512 T895 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 335516 T895 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 335935 T895 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 335944 T895 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 335946 T895 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 335957 T895 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 335961 T895 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 335964 T895 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 335965 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 335966 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 335967 T895 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 335967 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 335967 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 335968 T895 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 335968 T895 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/
[junit4:junit4] 2> 335968 T895 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 335969 T895 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 335970 T895 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2
[junit4:junit4] 2> 335970 T895 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index/
[junit4:junit4] 2> 335971 T895 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 335971 T895 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index
[junit4:junit4] 2> 335988 T895 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 335989 T895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 335991 T895 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 335991 T895 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 335992 T895 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 335992 T895 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 335993 T895 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 335993 T895 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 335993 T895 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 335994 T895 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 335994 T895 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 336005 T895 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 336011 T895 oass.SolrIndexSearcher.<init> Opening Searcher@3690524a main
[junit4:junit4] 2> 336012 T895 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 336012 T895 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 336016 T896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3690524a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 336017 T895 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 336017 T895 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15080 collection:collection1 shard:shard2
[junit4:junit4] 2> 336018 T895 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 336031 T895 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 336040 T895 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 336040 T895 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 336040 T895 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 336040 T895 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 336040 T895 oasc.SyncStrategy.syncToMe http://127.0.0.1:15080/collection1/ has no replicas
[junit4:junit4] 2> 336041 T895 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 336041 T895 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 337390 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 337407 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337407 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337407 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337407 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337434 T895 oasc.ZkController.register We are http://127.0.0.1:15080/collection1/ and leader is http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 337434 T895 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15080
[junit4:junit4] 2> 337434 T895 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 337434 T895 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 337435 T895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 337437 T895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 337437 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 337438 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 337438 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 337629 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 337632 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15100
[junit4:junit4] 2> 337632 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 337633 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 337633 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667
[junit4:junit4] 2> 337633 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/solr.xml
[junit4:junit4] 2> 337634 T836 oasc.CoreContainer.<init> New CoreContainer 230704970
[junit4:junit4] 2> 337634 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/'
[junit4:junit4] 2> 337635 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/'
[junit4:junit4] 2> 337734 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 337734 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 337735 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 337735 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 337735 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 337736 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 337736 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 337736 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 337737 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 337737 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 337748 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 337749 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 337749 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 337750 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 337753 T908 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25eb5807 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 337753 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 337765 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 337773 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 337775 T910 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e554da4 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 337776 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 337787 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 338790 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15100_
[junit4:junit4] 2> 338802 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15100_
[junit4:junit4] 2> 338806 T865 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T910 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T878 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338806 T894 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T858 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338807 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338807 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338807 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338814 T911 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 338814 T911 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 338922 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 338923 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15080__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15080_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15080"}
[junit4:junit4] 2> 338927 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15100_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15100"}
[junit4:junit4] 2> 338927 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 338927 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 338992 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 339818 T911 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1
[junit4:junit4] 2> 339818 T911 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 339819 T911 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 339819 T911 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 339821 T911 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/'
[junit4:junit4] 2> 339823 T911 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 339823 T911 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/lib/README' to classloader
[junit4:junit4] 2> 339873 T911 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 339929 T911 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 340030 T911 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 340037 T911 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 340622 T911 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 340631 T911 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 340633 T911 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 340644 T911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 340648 T911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 340651 T911 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 340652 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 340653 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 340653 T911 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 340654 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 340654 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 340654 T911 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 340655 T911 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/
[junit4:junit4] 2> 340655 T911 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 340655 T911 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 340656 T911 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3
[junit4:junit4] 2> 340657 T911 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index/
[junit4:junit4] 2> 340657 T911 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 340658 T911 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index
[junit4:junit4] 2> 340687 T911 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 340687 T911 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 340690 T911 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 340690 T911 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 340690 T911 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 340691 T911 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 340691 T911 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 340692 T911 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 340692 T911 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 340692 T911 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 340693 T911 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 340703 T911 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 340710 T911 oass.SolrIndexSearcher.<init> Opening Searcher@763a2bfc main
[junit4:junit4] 2> 340710 T911 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 340711 T911 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 340714 T912 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@763a2bfc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 340715 T911 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 340715 T911 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15100 collection:collection1 shard:shard3
[junit4:junit4] 2> 340716 T911 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 340730 T911 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 340741 T911 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 340741 T911 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 340742 T911 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 340742 T911 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 340742 T911 oasc.SyncStrategy.syncToMe http://127.0.0.1:15100/collection1/ has no replicas
[junit4:junit4] 2> 340742 T911 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 340742 T911 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 342027 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 342071 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342072 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342072 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342071 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342071 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342093 T911 oasc.ZkController.register We are http://127.0.0.1:15100/collection1/ and leader is http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 342093 T911 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15100
[junit4:junit4] 2> 342093 T911 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 342093 T911 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 342094 T911 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 342095 T911 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 342097 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 342098 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 342098 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 342323 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 342327 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15103
[junit4:junit4] 2> 342327 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 342327 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 342328 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328
[junit4:junit4] 2> 342328 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/solr.xml
[junit4:junit4] 2> 342328 T836 oasc.CoreContainer.<init> New CoreContainer 1197773348
[junit4:junit4] 2> 342329 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/'
[junit4:junit4] 2> 342329 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/'
[junit4:junit4] 2> 342414 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 342414 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 342415 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 342415 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 342415 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 342416 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 342416 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 342416 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 342417 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 342417 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 342429 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 342430 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 342430 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 342431 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 342434 T924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@321d96f7 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 342434 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 342437 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 342451 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 342455 T926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c341ed4 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 342456 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 342475 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 343479 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15103_
[junit4:junit4] 2> 343507 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15103_
[junit4:junit4] 2> 343511 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 343512 T894 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T858 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T865 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T926 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343512 T878 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343513 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343512 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343514 T910 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343513 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343525 T927 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 343525 T927 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 343586 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 343587 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15100__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15100_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15100"}
[junit4:junit4] 2> 343591 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15103_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15103"}
[junit4:junit4] 2> 343591 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 343592 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 343603 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 344532 T927 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1
[junit4:junit4] 2> 344532 T927 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 344533 T927 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 344533 T927 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 344535 T927 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/'
[junit4:junit4] 2> 344537 T927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 344537 T927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/lib/README' to classloader
[junit4:junit4] 2> 344591 T927 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 344667 T927 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 344768 T927 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 344775 T927 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 345388 T927 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 345400 T927 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 345405 T927 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 345420 T927 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 345425 T927 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 345430 T927 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 345431 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 345432 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 345433 T927 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 345434 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 345434 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 345434 T927 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 345435 T927 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/
[junit4:junit4] 2> 345435 T927 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 345436 T927 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 345437 T927 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4
[junit4:junit4] 2> 345438 T927 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index/
[junit4:junit4] 2> 345439 T927 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 345439 T927 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index
[junit4:junit4] 2> 345458 T927 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 345459 T927 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 345462 T927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 345462 T927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 345463 T927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 345464 T927 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 345465 T927 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 345465 T927 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 345465 T927 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 345466 T927 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 345466 T927 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 345484 T927 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 345495 T927 oass.SolrIndexSearcher.<init> Opening Searcher@2842f8db main
[junit4:junit4] 2> 345499 T927 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 345499 T927 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 345505 T928 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2842f8db main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 345507 T927 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 345507 T927 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15103 collection:collection1 shard:shard1
[junit4:junit4] 2> 345544 T927 oasc.ZkController.register We are http://127.0.0.1:15103/collection1/ and leader is http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 345544 T927 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15103
[junit4:junit4] 2> 345545 T927 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 345545 T927 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C195 name=collection1 org.apache.solr.core.SolrCore@15c25758 url=http://127.0.0.1:15103/collection1 node=127.0.0.1:15103_ C195_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:15103_, base_url=http://127.0.0.1:15103}
[junit4:junit4] 2> 345546 T929 C195 P15103 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 345546 T927 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 345546 T929 C195 P15103 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 345547 T929 C195 P15103 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 345547 T929 C195 P15103 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 345548 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 345548 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 345549 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 345549 T929 C195 P15103 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 345577 T868 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:15103__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 345846 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 345849 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15108
[junit4:junit4] 2> 345850 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 345851 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 345851 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792
[junit4:junit4] 2> 345852 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/solr.xml
[junit4:junit4] 2> 345852 T836 oasc.CoreContainer.<init> New CoreContainer 1379866054
[junit4:junit4] 2> 345853 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/'
[junit4:junit4] 2> 345853 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/'
[junit4:junit4] 2> 345983 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 345983 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 345984 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 345984 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 345985 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 345985 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 345986 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 345986 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 345987 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 345988 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 346005 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 346006 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 346006 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 346007 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 346012 T941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d245a44 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 346013 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 346022 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 346035 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 346037 T943 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a1cfe8e name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 346038 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 346049 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 346620 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 346621 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15103__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15103_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15103"}
[junit4:junit4] 2> 346642 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347054 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15108_
[junit4:junit4] 2> 347057 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15108_
[junit4:junit4] 2> 347059 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347060 T894 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T858 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T865 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T878 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347060 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347060 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347060 T926 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T943 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347062 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347061 T910 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347061 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347062 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347073 T944 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 347074 T944 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 347580 T868 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:15103__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 347580 T868 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:15103__collection1&state=recovering&nodeName=127.0.0.1:15103_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 348149 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 348149 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15108_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15108"}
[junit4:junit4] 2> 348150 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 348150 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 348179 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 349079 T944 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1
[junit4:junit4] 2> 349079 T944 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 349080 T944 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 349080 T944 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 349082 T944 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/'
[junit4:junit4] 2> 349084 T944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/lib/README' to classloader
[junit4:junit4] 2> 349084 T944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 349136 T944 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 349198 T944 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 349200 T944 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 349206 T944 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C196 name=collection1 org.apache.solr.core.SolrCore@15c25758 url=http://127.0.0.1:15103/collection1 node=127.0.0.1:15103_ C196_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:15103_, base_url=http://127.0.0.1:15103}
[junit4:junit4] 2> 349583 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53832/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 349584 T929 C196 P15103 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:15103 START replicas=[http://127.0.0.1:53832/collection1/] nUpdates=100
[junit4:junit4] 2> 349586 T929 C196 P15103 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 349589 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 349589 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 349590 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 349590 T929 C196 P15103 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 349591 T929 C196 P15103 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53832/collection1/. core=collection1
[junit4:junit4] 2> 349591 T929 C196 P15103 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C197 name=collection1 org.apache.solr.core.SolrCore@4d5898ce url=http://127.0.0.1:53832/collection1 node=127.0.0.1:53832_ C197_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53832_, base_url=http://127.0.0.1:53832, leader=true}
[junit4:junit4] 2> 349617 T869 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 349620 T870 C197 P53832 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 349635 T870 C197 P53832 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 349639 T870 C197 P53832 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 349649 T870 C197 P53832 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 349650 T870 C197 P53832 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 349652 T870 C197 P53832 oass.SolrIndexSearcher.<init> Opening Searcher@7c7284f4 realtime
[junit4:junit4] 2> 349652 T870 C197 P53832 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 349653 T870 C197 P53832 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 35
[junit4:junit4] 2> 349654 T929 C196 P15103 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 349654 T929 C196 P15103 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 349658 T871 C197 P53832 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 349659 T871 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
[junit4:junit4] 2> 349660 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 349660 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 349660 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 349664 T871 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 349666 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 349668 T929 C196 P15103 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884
[junit4:junit4] 2> 349669 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfe4b75 fullCopy=false
[junit4:junit4] 2> 349674 T871 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 349677 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 349745 T929 C196 P15103 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 349746 T929 C196 P15103 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 349752 T929 C196 P15103 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 349754 T929 C196 P15103 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 349755 T929 C196 P15103 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 349756 T929 C196 P15103 oass.SolrIndexSearcher.<init> Opening Searcher@38032b2b main
[junit4:junit4] 2> 349758 T928 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38032b2b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 349758 T929 C196 P15103 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884;done=true>>]
[junit4:junit4] 2> 349759 T929 C196 P15103 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884
[junit4:junit4] 2> 349759 T929 C196 P15103 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884
[junit4:junit4] 2> 349760 T929 C196 P15103 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 349760 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 349761 T929 C196 P15103 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 349761 T929 C196 P15103 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 349763 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 349953 T944 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 349965 T944 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 349969 T944 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 349984 T944 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 350003 T944 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 350007 T944 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 350008 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 350009 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 350009 T944 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 350010 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 350011 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 350011 T944 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 350011 T944 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/
[junit4:junit4] 2> 350012 T944 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 350012 T944 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 350013 T944 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5
[junit4:junit4] 2> 350014 T944 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index/
[junit4:junit4] 2> 350015 T944 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 350016 T944 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index
[junit4:junit4] 2> 350021 T944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 350022 T944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 350025 T944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 350025 T944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 350026 T944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 350026 T944 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 350027 T944 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 350027 T944 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 350031 T944 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 350031 T944 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 350032 T944 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 350048 T944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 350058 T944 oass.SolrIndexSearcher.<init> Opening Searcher@3b890dc0 main
[junit4:junit4] 2> 350059 T944 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 350059 T944 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 350064 T947 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b890dc0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 350066 T944 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 350066 T944 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15108 collection:collection1 shard:shard2
[junit4:junit4] 2> 350070 T944 oasc.ZkController.register We are http://127.0.0.1:15108/collection1/ and leader is http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 350070 T944 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15108
[junit4:junit4] 2> 350071 T944 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 350071 T944 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C198 name=collection1 org.apache.solr.core.SolrCore@33a2c26b url=http://127.0.0.1:15108/collection1 node=127.0.0.1:15108_ C198_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:15108_, base_url=http://127.0.0.1:15108}
[junit4:junit4] 2> 350071 T948 C198 P15108 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 350072 T948 C198 P15108 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 350072 T944 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 350072 T948 C198 P15108 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 350073 T948 C198 P15108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 350073 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 350074 T948 C198 P15108 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 350074 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 350075 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 350099 T884 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:15108__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 350368 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 350373 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15115
[junit4:junit4] 2> 350373 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 350374 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 350375 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315
[junit4:junit4] 2> 350375 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/solr.xml
[junit4:junit4] 2> 350376 T836 oasc.CoreContainer.<init> New CoreContainer 1431465309
[junit4:junit4] 2> 350376 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/'
[junit4:junit4] 2> 350377 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/'
[junit4:junit4] 2> 350509 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 350510 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 350511 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 350511 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 350512 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 350512 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 350513 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 350513 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 350514 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 350514 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 350532 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 350533 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 350533 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 350534 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 350537 T960 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37189f9d name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 350538 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 350550 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 350562 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 350564 T962 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bd803 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 350564 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 350575 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 351194 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 351195 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15103__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15103_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15103"}
[junit4:junit4] 2> 351226 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15108__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15108_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15108"}
[junit4:junit4] 2> 351248 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351579 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15115_
[junit4:junit4] 2> 351582 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15115_
[junit4:junit4] 2> 351584 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351585 T926 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T894 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T943 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T962 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351585 T858 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T878 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T865 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351586 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351586 T910 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351586 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351586 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351585 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351585 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351587 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351599 T963 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 351600 T963 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 352102 T884 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:15108__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 352102 T884 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:15108__collection1&state=recovering&nodeName=127.0.0.1:15108_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 352756 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 352757 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15115_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15115"}
[junit4:junit4] 2> 352757 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 352757 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 352772 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 353605 T963 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1
[junit4:junit4] 2> 353605 T963 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 353606 T963 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 353606 T963 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 353608 T963 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/'
[junit4:junit4] 2> 353609 T963 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/lib/README' to classloader
[junit4:junit4] 2> 353609 T963 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 353646 T963 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 353689 T963 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 353690 T963 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 353695 T963 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C199 name=collection1 org.apache.solr.core.SolrCore@33a2c26b url=http://127.0.0.1:15108/collection1 node=127.0.0.1:15108_ C199_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:15108_, base_url=http://127.0.0.1:15108}
[junit4:junit4] 2> 354104 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:15080/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 354104 T948 C199 P15108 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:15108 START replicas=[http://127.0.0.1:15080/collection1/] nUpdates=100
[junit4:junit4] 2> 354105 T948 C199 P15108 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 354105 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 354105 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 354105 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 354106 T948 C199 P15108 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 354106 T948 C199 P15108 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:15080/collection1/. core=collection1
[junit4:junit4] 2> 354106 T948 C199 P15108 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C200 name=collection1 org.apache.solr.core.SolrCore@513b1428 url=http://127.0.0.1:15080/collection1 node=127.0.0.1:15080_ C200_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:15080_, base_url=http://127.0.0.1:15080, leader=true}
[junit4:junit4] 2> 354115 T885 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 354119 T886 C200 P15080 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 354125 T886 C200 P15080 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 354125 T886 C200 P15080 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 354171 T886 C200 P15080 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 354172 T886 C200 P15080 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 354174 T886 C200 P15080 oass.SolrIndexSearcher.<init> Opening Searcher@7ccd6ee1 realtime
[junit4:junit4] 2> 354175 T886 C200 P15080 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 354175 T886 C200 P15080 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 56
[junit4:junit4] 2> 354176 T948 C199 P15108 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 354177 T948 C199 P15108 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 354179 T887 C200 P15080 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 354180 T887 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 354180 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 354181 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 354181 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 354184 T887 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 354185 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 354187 T948 C199 P15108 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404
[junit4:junit4] 2> 354188 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5350c62a fullCopy=false
[junit4:junit4] 2> 354192 T887 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 354194 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 354197 T963 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 354210 T963 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 354214 T963 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 354222 T948 C199 P15108 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 354222 T948 C199 P15108 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 354227 T948 C199 P15108 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 354228 T948 C199 P15108 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 354229 T948 C199 P15108 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 354229 T963 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 354230 T948 C199 P15108 oass.SolrIndexSearcher.<init> Opening Searcher@193d71d4 main
[junit4:junit4] 2> 354231 T947 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@193d71d4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 354232 T948 C199 P15108 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404;done=true>>]
[junit4:junit4] 2> 354232 T948 C199 P15108 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404
[junit4:junit4] 2> 354232 T948 C199 P15108 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 354236 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 354255 T963 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 354262 T963 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 354264 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 354264 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 354265 T963 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 354266 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 354266 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 354266 T963 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 354267 T963 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/
[junit4:junit4] 2> 354267 T963 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 354268 T963 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 354269 T963 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6
[junit4:junit4] 2> 354270 T963 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index/
[junit4:junit4] 2> 354271 T963 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 354272 T963 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index
[junit4:junit4] 2> 354278 T963 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 354278 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 354278 T963 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 354279 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15108__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15108_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15108"}
[junit4:junit4] 2> 354282 T963 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 354282 T963 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 354283 T963 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 354285 T963 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 354286 T963 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 354286 T963 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 354290 T963 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 354290 T963 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 354291 T963 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 354299 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354300 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354308 T963 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 354317 T963 oass.SolrIndexSearcher.<init> Opening Searcher@68058dec main
[junit4:junit4] 2> 354319 T963 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 354319 T963 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 354324 T966 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@68058dec main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 354326 T963 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 354327 T963 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15115 collection:collection1 shard:shard3
[junit4:junit4] 2> 354331 T963 oasc.ZkController.register We are http://127.0.0.1:15115/collection1/ and leader is http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 354332 T963 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15115
[junit4:junit4] 2> 354332 T963 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 354332 T963 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C201 name=collection1 org.apache.solr.core.SolrCore@13e227a5 url=http://127.0.0.1:15115/collection1 node=127.0.0.1:15115_ C201_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:15115_, base_url=http://127.0.0.1:15115}
[junit4:junit4] 2> 354333 T967 C201 P15115 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 354333 T963 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 354333 T967 C201 P15115 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 354334 T967 C201 P15115 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 354334 T967 C201 P15115 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 354335 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 354335 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 354336 T967 C201 P15115 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 354336 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 354356 T900 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:15115__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 354624 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 354628 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15122
[junit4:junit4] 2> 354629 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 354629 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 354630 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572
[junit4:junit4] 2> 354630 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/solr.xml
[junit4:junit4] 2> 354631 T836 oasc.CoreContainer.<init> New CoreContainer 1851212974
[junit4:junit4] 2> 354631 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/'
[junit4:junit4] 2> 354632 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/'
[junit4:junit4] 2> 354762 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 354763 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 354763 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 354764 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 354764 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 354765 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 354766 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 354766 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 354767 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 354767 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 354783 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 354784 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 354784 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 354785 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 354808 T979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42f91b74 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 354809 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 354812 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 354823 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 354825 T981 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a7367f name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 354825 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 354844 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 355804 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 355805 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15115__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15115_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15115"}
[junit4:junit4] 2> 355844 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355848 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15122_
[junit4:junit4] 2> 355850 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15122_
[junit4:junit4] 2> 355858 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355858 T926 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T943 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T858 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T962 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T894 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355859 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355859 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355859 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355858 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355858 T878 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T865 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355861 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355860 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355860 T910 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355861 T981 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355866 T982 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 355866 T982 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 356361 T900 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:15115__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 356361 T900 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:15115__collection1&state=recovering&nodeName=127.0.0.1:15115_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2005
[junit4:junit4] 2> 357352 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 357353 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15122_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15122"}
[junit4:junit4] 2> 357353 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 357353 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 357369 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357370 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357870 T982 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1
[junit4:junit4] 2> 357870 T982 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 357871 T982 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 357871 T982 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 357873 T982 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1/'
[junit4:junit4] 2> 357875 T982 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 357875 T982 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1/lib/README' to classloader
[junit4:junit4] 2> 357930 T982 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 357990 T982 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2
[...truncated too long message...]
617 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:20586 DONE. sync succeeded
[junit4:junit4] 2> 1546434 T2617 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 1546434 T2617 oasc.ShardLeaderElectionContext.runLeaderProcess Sync was not a success but no one else is active! I am the leader
[junit4:junit4] 2> 1546434 T2617 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20586/collection1/
[junit4:junit4] 2> 1546435 T2617 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@408211a1
[junit4:junit4] 2> 1546442 T2617 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=85,cumulative_deletesById=30,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 1546442 T2617 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1546442 T2617 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1546443 T2617 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1546470 T2617 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2/index;done=false>>]
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2/index
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2;done=false>>]
[junit4:junit4] 2> 1546472 T2617 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2
[junit4:junit4] 2> 1546472 T2617 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1546472 T2617 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/89867583342182410-127.0.0.1:20586__collection1-n_0000000002
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:260)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1546473 T2617 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1546473 T2617 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1546474 T2617 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1546474 T2617 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=SimpleText), text=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=SimpleText), rnd_b=PostingsFormat(name=Direct), intDefault=PostingsFormat(name=SimpleText), timestamp=PostingsFormat(name=SimpleText), id=PostingsFormat(name=Direct), a_t=PostingsFormat(name=SimpleText), range_facet_sl=PostingsFormat(name=MockRandom), range_facet_si=MockFixedIntBlock(blockSize=325), other_tl1=PostingsFormat(name=SimpleText), multiDefault=MockFixedIntBlock(blockSize=325), a_si=MockFixedIntBlock(blockSize=325)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=nl_NL, timezone=Australia/Broken_Hill
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=152398288,total=321912832
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestStressRecovery, TestFastLRUCache, WordBreakSolrSpellCheckerTest, QueryElevationComponentTest, ShowFileRequestHandlerTest, SearchHandlerTest, FileBasedSpellCheckerTest, SimpleFacetsTest, TestConfig, DocValuesTest, CircularListTest, ShardRoutingTest, SliceStateTest, RegexBoostProcessorTest, PluginInfoTest, TestFieldCollectionResource, TestDocSet, TestSolrIndexConfig, ZkControllerTest, BasicDistributedZkTest, ZkCLITest, BasicFunctionalityTest, CacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, SuggesterFSTTest, SolrInfoMBeanTest, JsonLoaderTest, PeerSyncTest, QueryResultKeyTest, StatelessScriptUpdateProcessorFactoryTest, OverseerTest, MBeansHandlerTest, DocumentBuilderTest, TestDynamicFieldResource, TestPluginEnable, SpatialFilterTest, DirectUpdateHandlerOptimizeTest, ChaosMonkeyNothingIsSafeTest, TestFunctionQuery, PreAnalyzedFieldTest, TestLFUCache, TestCharFilters, TestMaxScoreQueryParser, TestPhraseSuggestions, DistributedQueryElevationComponentTest, TestRealTimeGet, TestRemoteStreaming, ShardSplitTest, CSVRequestHandlerTest, TestSurroundQueryParser, TestQueryUtils, SolrTestCaseJ4Test, TestCoreContainer, RAMDirectoryFactoryTest, TestReversedWildcardFilterFactory, QueryEqualityTest, TestXIncludeConfig, PolyFieldTest, SOLR749Test, UnloadDistributedZkTest, TestJoin, TestSchemaResource, IndexSchemaTest, BadComponentTest, FastVectorHighlighterTest, TestPHPSerializedResponseWriter, TestRandomDVFaceting, TestWriterPerf, SolrCoreTest, SyncSliceTest, RecoveryZkTest]
[junit4:junit4] Completed on J0 in 144.77s, 1 test, 1 failure <<< FAILURES!
[...truncated 440 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:385: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:365: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 2 failures, 13 ignored (7 assumptions)
Total time: 56 minutes 6 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4069 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4069/
2 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<210> but was:<42>
Stack Trace:
java.lang.AssertionError: expected:<210> but was:<42>
at __randomizedtesting.SeedInfo.seed([9D654810402FC7CB:1C83C6083770A7F7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:49730/hq/qf returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:49730/hq/qf returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([9D654810402FC7CB:1C83C6083770A7F7]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9207 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 162152 T237 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /rhm/w
[junit4:junit4] 2> 162158 T237 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371335366311
[junit4:junit4] 2> 162160 T237 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 162160 T238 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 162261 T237 oasc.ZkTestServer.run start zk server on port:30938
[junit4:junit4] 2> 162262 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162267 T244 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ac43d60 name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162267 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162268 T237 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 162278 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162281 T246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3a9ea0 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162281 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162281 T237 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 162293 T237 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 162296 T237 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 162299 T237 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 162303 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 162304 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 162316 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 162317 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 162421 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 162422 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 162426 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 162427 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 162435 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 162436 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 162440 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 162440 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 162444 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 162445 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 162449 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 162449 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 162453 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 162454 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 162689 T237 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 162695 T237 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30941
[junit4:junit4] 2> 162695 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 162696 T237 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 162696 T237 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611
[junit4:junit4] 2> 162697 T237 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/solr.xml
[junit4:junit4] 2> 162697 T237 oasc.CoreContainer.<init> New CoreContainer 2087423458
[junit4:junit4] 2> 162698 T237 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/'
[junit4:junit4] 2> 162699 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/'
[junit4:junit4] 2> 162816 T237 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 162817 T237 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 162817 T237 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 162818 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 162818 T237 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 162818 T237 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 162819 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 162819 T237 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 162820 T237 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 162820 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 162837 T237 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 162838 T237 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30938/solr
[junit4:junit4] 2> 162838 T237 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 162840 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162843 T257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2317549f name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162844 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162867 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 162881 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162885 T259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd22554 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162885 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162888 T237 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 162899 T237 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 162903 T237 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 162906 T237 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30941_rhm%2Fw
[junit4:junit4] 2> 162908 T237 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30941_rhm%2Fw
[junit4:junit4] 2> 162918 T237 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 162933 T237 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 162937 T237 oasc.Overseer.start Overseer (id=89871834567016451-127.0.0.1:30941_rhm%2Fw-n_0000000000) starting
[junit4:junit4] 2> 162947 T237 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 162957 T261 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 162958 T237 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 162962 T237 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 162964 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 162968 T260 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 162972 T262 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 162972 T262 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 164473 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 164474 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30941_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30941/rhm/w"}
[junit4:junit4] 2> 164475 T260 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 164475 T260 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 164497 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 164976 T262 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1
[junit4:junit4] 2> 164976 T262 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 164977 T262 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 164977 T262 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 164979 T262 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/'
[junit4:junit4] 2> 164981 T262 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/lib/README' to classloader
[junit4:junit4] 2> 164982 T262 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 165038 T262 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 165107 T262 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 165209 T262 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 165218 T262 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 165617 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
[junit4:junit4] 2> 165885 T262 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 165899 T262 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 165903 T262 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 165931 T262 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 165936 T262 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 165941 T262 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 165943 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 165943 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 165943 T262 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 165945 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 165945 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 165946 T262 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 165946 T262 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/
[junit4:junit4] 2> 165946 T262 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@69fba118
[junit4:junit4] 2> 165947 T262 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 165948 T262 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data
[junit4:junit4] 2> 165948 T262 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/index/
[junit4:junit4] 2> 165948 T262 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 165949 T262 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/index
[junit4:junit4] 2> 165954 T262 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@43730ad7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@504182ad; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 165955 T262 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 165958 T262 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 165958 T262 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 165959 T262 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 165960 T262 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 165960 T262 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 165960 T262 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 165961 T262 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 165961 T262 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 165962 T262 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 165977 T262 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 165984 T262 oass.SolrIndexSearcher.<init> Opening Searcher@5fa05dda main
[junit4:junit4] 2> 165985 T262 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/tlog
[junit4:junit4] 2> 165986 T262 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 165986 T262 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 165990 T263 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5fa05dda main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 165993 T262 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 165993 T262 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30941/rhm/w collection:control_collection shard:shard1
[junit4:junit4] 2> 165994 T262 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 166014 T262 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 166025 T262 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 166026 T262 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 166026 T262 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:30941/rhm/w/collection1/
[junit4:junit4] 2> 166026 T262 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 166027 T262 oasc.SyncStrategy.syncToMe http://127.0.0.1:30941/rhm/w/collection1/ has no replicas
[junit4:junit4] 2> 166027 T262 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:30941/rhm/w/collection1/
[junit4:junit4] 2> 166027 T262 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 167516 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 167536 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 167565 T262 oasc.ZkController.register We are http://127.0.0.1:30941/rhm/w/collection1/ and leader is http://127.0.0.1:30941/rhm/w/collection1/
[junit4:junit4] 2> 167565 T262 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30941/rhm/w
[junit4:junit4] 2> 167565 T262 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 167566 T262 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 167566 T262 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 167575 T262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 167576 T237 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 167577 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 167578 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 167598 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 167600 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 167602 T266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3df4fd50 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 167603 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 167605 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 167608 T237 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 167877 T237 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 167880 T237 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30946
[junit4:junit4] 2> 167881 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 167882 T237 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 167882 T237 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760
[junit4:junit4] 2> 167883 T237 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/solr.xml
[junit4:junit4] 2> 167883 T237 oasc.CoreContainer.<init> New CoreContainer 260157985
[junit4:junit4] 2> 167884 T237 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/'
[junit4:junit4] 2> 167885 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/'
[junit4:junit4] 2> 168018 T237 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 168019 T237 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 168019 T237 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 168020 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 168020 T237 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 168021 T237 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 168021 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 168022 T237 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 168022 T237 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 168023 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 168041 T237 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 168041 T237 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30938/solr
[junit4:junit4] 2> 168042 T237 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 168043 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 168054 T277 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51c5f268 name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 168055 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 168069 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 168082 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 168085 T279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a3f8ce2 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 168085 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 168091 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 169043 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 169044 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30941_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30941/rhm/w"}
[junit4:junit4] 2> 169066 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169066 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169066 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169095 T237 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30946_rhm%2Fw
[junit4:junit4] 2> 169098 T237 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30946_rhm%2Fw
[junit4:junit4] 2> 169100 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169101 T259 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 169101 T279 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 169101 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 169102 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 169102 T266 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 169115 T280 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 169116 T280 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 170573 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 170574 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30946_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30946/rhm/w"}
[junit4:junit4] 2> 170575 T260 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 170575 T260 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 170589 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 170589 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 170589 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 171119 T280 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1
[junit4:junit4] 2> 171119 T280 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 171120 T280 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 171120 T280 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 171122 T280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/'
[junit4:junit4] 2> 171124 T280 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 171125 T280 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/lib/README' to classloader
[junit4:junit4] 2> 171179 T280 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 171242 T280 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 171344 T280 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 171353 T280 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 171957 T280 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 171967 T280 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 171970 T280 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 171992 T280 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 171996 T280 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 172000 T280 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 172001 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 172001 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 172002 T280 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 172003 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 172003 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 172004 T280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 172004 T280 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/
[junit4:junit4] 2> 172004 T280 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@69fba118
[junit4:junit4] 2> 172005 T280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 172005 T280 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1
[junit4:junit4] 2> 172005 T280 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/index/
[junit4:junit4] 2> 172005 T280 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 172006 T280 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/index
[junit4:junit4] 2> 172009 T280 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 172009 T280 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 172012 T280 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 172012 T280 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 172012 T280 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 172013 T280 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 172013 T280 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 172014 T280 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 172014 T280 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 172015 T280 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 172015 T280 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 172026 T280 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 172031 T280 oass.SolrIndexSearcher.<init> Opening Searcher@6beb12d6 main
[junit4:junit4] 2> 172031 T280 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/tlog
[junit4:junit4] 2> 172032 T280 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 172032 T280 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 172036 T281 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6beb12d6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 172038 T280 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 172038 T280 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30946/rhm/w collection:collection1 shard:shard1
[junit4:junit4] 2> 172039 T280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 172058 T280 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 172062 T280 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 172063 T280 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 172063 T280 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 172063 T280 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 172064 T280 oasc.SyncStrategy.syncToMe http://127.0.0.1:30946/rhm/w/collection1/ has no replicas
[junit4:junit4] 2> 172064 T280 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 172064 T280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 172094 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 172111 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 172111 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 172111 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 172130 T280 oasc.ZkController.register We are http://127.0.0.1:30946/rhm/w/collection1/ and leader is http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 172130 T280 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30946/rhm/w
[junit4:junit4] 2> 172130 T280 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 172130 T280 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 172131 T280 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 172133 T280 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 172134 T237 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 172134 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 172135 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 172319 T237 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 172321 T237 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30949
[junit4:junit4] 2> 172322 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 172322 T237 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 172323 T237 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299
[junit4:junit4] 2> 172323 T237 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/solr.xml
[junit4:junit4] 2> 172323 T237 oasc.CoreContainer.<init> New CoreContainer 262392533
[junit4:junit4] 2> 172324 T237 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/'
[junit4:junit4] 2> 172324 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/'
[junit4:junit4] 2> 172425 T237 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 172425 T237 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 172425 T237 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 172426 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 172426 T237 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 172427 T237 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 172427 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 172427 T237 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 172428 T237 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 172428 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 172440 T237 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 172441 T237 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30938/solr
[junit4:junit4] 2> 172441 T237 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 172442 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 172445 T293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@781c4dc3 name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 172446 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 172449 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 172457 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 172460 T295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5360b61f name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 172460 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 172465 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 173468 T237 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30949_rhm%2Fw
[junit4:junit4] 2> 173497 T237 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30949_rhm%2Fw
[junit4:junit4] 2> 173500 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 173501 T279 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173501 T295 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173501 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173501 T259 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173502 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173503 T266 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173510 T296 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 173510 T296 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 173625 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 173626 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30946_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30946/rhm/w"}
[junit4:junit4] 2> 173660 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 173661 T260 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 173661 T260 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 173676 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173676 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173676 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173676 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 174518 T296 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1
[junit4:junit4] 2> 174518 T296 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 174519 T296 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 174520 T296 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 174522 T296 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/'
[junit4:junit4] 2> 174523 T296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 174524 T296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/lib/README' to classloader
[junit4:junit4] 2> 174578 T296 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 174641 T296 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 174742 T296 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 174754 T296 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 175390 T296 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 175400 T296 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 175403 T296 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 175424 T296 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 175429 T296 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 175433 T296 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 175434 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 175435 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 175435 T296 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 175436 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 175436 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 175437 T296 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 175437 T296 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/
[junit4:junit4] 2> 175437 T296 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@69fba118
[junit4:junit4] 2> 175438 T296 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 175438 T296 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2
[junit4:junit4] 2> 175438 T296 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index/
[junit4:junit4] 2> 175439 T296 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 175439 T296 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index
[junit4:junit4] 2> 175442 T296 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 175442 T296 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 175444 T296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 175445 T296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 175445 T296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 175446 T296 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 175446 T296 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 175447 T296 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 175447 T296 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 175447 T296 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 175448 T296 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 175458 T296 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 175464 T296 oass.SolrIndexSearcher.<init> Opening Searcher@4be04bd0 main
[junit4:junit4] 2> 175464 T296 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/tlog
[junit4:junit4] 2> 175465 T296 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 175465 T296 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 175469 T297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4be04bd0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 175471 T296 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 175471 T296 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30949/rhm/w collection:collection1 shard:shard1
[junit4:junit4] 2> 175500 T296 oasc.ZkController.register We are http://127.0.0.1:30949/rhm/w/collection1/ and leader is http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 175500 T296 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30949/rhm/w
[junit4:junit4] 2> 175501 T296 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 175501 T296 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C109 name=collection1 org.apache.solr.core.SolrCore@a41ba9d url=http://127.0.0.1:30949/rhm/w/collection1 node=127.0.0.1:30949_rhm%2Fw C109_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:30949_rhm%2Fw, base_url=http://127.0.0.1:30949/rhm/w}
[junit4:junit4] 2> 175502 T298 C109 P30949 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 175502 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 175502 T296 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 175502 T298 C109 P30949 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 175503 T298 C109 P30949 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 175503 T237 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 175504 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 175504 T298 C109 P30949 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 175504 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 175519 T237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 175520 T269 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C110 name=collection1 org.apache.solr.core.SolrCore@33287c71 url=http://127.0.0.1:30941/rhm/w/collection1 node=127.0.0.1:30941_rhm%2Fw C110_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:30941_rhm%2Fw, base_url=http://127.0.0.1:30941/rhm/w, leader=true}
[junit4:junit4] 2> 175532 T250 C110 P30941 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@43730ad7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@504182ad; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 175532 T250 C110 P30941 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 175537 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[1 (1437949367085432832)]} 0 8
[junit4:junit4] 2> 175539 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10001 (1437949367087529984)]} 0 7
[junit4:junit4] 2> ASYNC NEW_CORE C111 name=collection1 org.apache.solr.core.SolrCore@5e1f92d8 url=http://127.0.0.1:30946/rhm/w/collection1 node=127.0.0.1:30946_rhm%2Fw C111_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:30946_rhm%2Fw, base_url=http://127.0.0.1:30946/rhm/w, leader=true}
[junit4:junit4] 2> 175556 T271 C111 P30946 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 175556 T271 C111 P30946 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 175560 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10001 (1437949367109550080)]} 0 7
[junit4:junit4] 2> 175561 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[1 (1437949367112695808)]} 0 6
[junit4:junit4] 2> 175657 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10002 (1437949367216504832)]} 0 3
[junit4:junit4] 2> 175658 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[2 (1437949367218601984)]} 0 2
[junit4:junit4] 2> 175662 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10002 (1437949367221747712)]} 0 2
[junit4:junit4] 2> 175663 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[2 (1437949367222796288)]} 0 2
[junit4:junit4] 2> 175706 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10003 (1437949367267885056)]} 0 2
[junit4:junit4] 2> 175708 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[3 (1437949367269982208)]} 0 3
[junit4:junit4] 2> 175711 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10003 (1437949367274176512)]} 0 2
[junit4:junit4] 2> 175714 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[3 (1437949367276273664)]} 0 3
[junit4:junit4] 2> 175805 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10001 (-1437949367373791232)]} 0 1
[junit4:junit4] 2> 175809 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[1 (-1437949367377985536)]} 0 1
[junit4:junit4] 2> 175809 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10001 (-1437949367377985536)]} 0 1
[junit4:junit4] 2> 175813 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[1 (-1437949367381131264)]} 0 1
[junit4:junit4] 2> 175814 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10004 (1437949367382179840)]} 0 2
[junit4:junit4] 2> 175817 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[4 (1437949367384276992)]} 0 2
[junit4:junit4] 2> 175819 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10004 (1437949367387422720)]} 0 2
[junit4:junit4] 2> 175822 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[4 (1437949367389519872)]} 0 3
[junit4:junit4] 2> 175827 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10005 (1437949367394762752)]} 0 3
[junit4:junit4] 2> 175830 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[5 (1437949367398957056)]} 0 2
[junit4:junit4] 2> 175833 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10005 (1437949367401054208)]} 0 3
[junit4:junit4] 2> 175835 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[5 (1437949367403151360)]} 0 2
[junit4:junit4] 2> 175922 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10006 (1437949367494377472)]} 0 2
[junit4:junit4] 2> 175923 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[6 (1437949367495426048)]} 0 3
[junit4:junit4] 2> 175927 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10006 (1437949367500668928)]} 0 2
[junit4:junit4] 2> 175929 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[6 (1437949367501717504)]} 0 3
[junit4:junit4] 2> 175948 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10002 (-1437949367523737600)]} 0 1
[junit4:junit4] 2> 175950 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[2 (-1437949367525834752)]} 0 1
[junit4:junit4] 2> 175953 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10002 (-1437949367528980480)]} 0 1
[junit4:junit4] 2> 175954 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[2 (-1437949367528980481)]} 0 1
[junit4:junit4] 2> 175959 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[7 (1437949367533174784)]} 0 3
[junit4:junit4] 2> 175960 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10007 (1437949367533174785)]} 0 3
[junit4:junit4] 2> 175965 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[7 (1437949367539466240)]} 0 3
[junit4:junit4] 2> 175967 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10007 (1437949367541563392)]} 0 3
[junit4:junit4] 2> 176058 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[4 (-1437949367638032384)]} 0 1
[junit4:junit4] 2> 176059 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10004 (-1437949367640129536)]} 0 1
[junit4:junit4] 2> 176062 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[4 (-1437949367643275264)]} 0 1
[junit4:junit4] 2> 176063 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10004 (-1437949367643275265)]} 0 1
[junit4:junit4] 2> 176068 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[8 (1437949367647469568)]} 0 3
[junit4:junit4] 2> 176069 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10008 (1437949367648518144)]} 0 3
[junit4:junit4] 2> 176074 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[8 (1437949367653761024)]} 0 3
[junit4:junit4] 2> 176075 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10008 (1437949367654809600)]} 0 3
[junit4:junit4] 2> 176102 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[9 (1437949367684169728)]} 0 2
[junit4:junit4] 2> 176104 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10009 (1437949367686266880)]} 0 2
[junit4:junit4] 2> 176107 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[9 (1437949367689412608)]} 0 2
[junit4:junit4] 2> 176109 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10009 (1437949367691509760)]} 0 2
[junit4:junit4] 2> 176138 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10 (1437949367720869888)]} 0 3
[junit4:junit4] 2> 176139 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10010 (1437949367722967040)]} 0 2
[junit4:junit4] 2> 176143 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10 (1437949367727161344)]} 0 2
[junit4:junit4] 2> 176144 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10010 (1437949367728209920)]} 0 2
[junit4:junit4] 2> 176147 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[6 (-1437949367732404224)]} 0 1
[junit4:junit4] 2> 176148 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10006 (-1437949367733452800)]} 0 1
[junit4:junit4] 2> 176151 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[6 (-1437949367735549952)]} 0 1
[junit4:junit4] 2> 176151 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10006 (-1437949367736598528)]} 0 0
[junit4:junit4] 2> 176155 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[11 (1437949367738695680)]} 0 2
[junit4:junit4] 2> 176156 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10011 (1437949367739744256)]} 0 2
[junit4:junit4] 2> 176160 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[11 (1437949367743938560)]} 0 2
[junit4:junit4] 2> 176161 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10011 (1437949367744987136)]} 0 2
[junit4:junit4] 2> 176188 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[7 (-1437949367774347264)]} 0 1
[junit4:junit4] 2> 176189 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10007 (-1437949367775395840)]} 0 1
[junit4:junit4] 2> 176190 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[7 (-1437949367777492992)]} 0 0
[junit4:junit4] 2> 176191 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10007 (-1437949367778541568)]} 0 0
[junit4:junit4] 2> 176195 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[12 (1437949367780638720)]} 0 2
[junit4:junit4] 2> 176196 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10012 (1437949367781687296)]} 0 2
[junit4:junit4] 2> 176198 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[12 (1437949367784833024)]} 0 1
[junit4:junit4] 2> 176200 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10012 (1437949367785881600)]} 0 2
[junit4:junit4] 2> 176256 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[13 (1437949367845650432)]} 0 2
[junit4:junit4] 2> 176257 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10013 (1437949367845650433)]} 0 2
[junit4:junit4] 2> 176261 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10013 (1437949367849844736)]} 0 2
[junit4:junit4] 2> 176261 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[13 (1437949367849844737)]} 0 2
[junit4:junit4] 2> 176340 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10014 (1437949367933730816)]} 0 2
[junit4:junit4] 2> 176341 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[14 (1437949367934779392)]} 0 2
[junit4:junit4] 2> 176344 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[14 (1437949367937925120)]} 0 1
[junit4:junit4] 2> 176345 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10014 (1437949367938973696)]} 0 2
[junit4:junit4] 2> 176427 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[15 (1437949368023908352)]} 0 2
[junit4:junit4] 2> 176428 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10015 (1437949368024956928)]} 0 3
[junit4:junit4] 2> 176433 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[15 (1437949368030199808)]} 0 3
[junit4:junit4] 2> 176434 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10015 (1437949368031248384)]} 0 2
[junit4:junit4] 2> 176535 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[16 (1437949368137154560)]} 0 3
[junit4:junit4] 2> 176537 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10016 (1437949368138203136)]} 0 4
[junit4:junit4] 2> 176541 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[16 (1437949368144494592)]} 0 2
[junit4:junit4] 2> 176543 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10016 (1437949368145543168)]} 0 3
[junit4:junit4] 2> 176619 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[9 (-1437949368227332096)]} 0 1
[junit4:junit4] 2> 176620 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10009 (-1437949368228380672)]} 0 1
[junit4:junit4] 2> 176623 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[9 (-1437949368231526400)]} 0 1
[junit4:junit4] 2> 176624 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10009 (-1437949368231526401)]} 0 1
[junit4:junit4] 2> 176628 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[17 (1437949368235720704)]} 0 2
[junit4:junit4] 2> 176630 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10017 (1437949368236769280)]} 0 3
[junit4:junit4] 2> 176633 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[17 (1437949368240963584)]} 0 2
[junit4:junit4] 2> 176636 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10017 (1437949368243060736)]} 0 3
[junit4:junit4] 2> 176648 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[15 (-1437949368257740800)]} 0 1
[junit4:junit4] 2> 176651 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10015 (-1437949368259837952)]} 0 1
[junit4:junit4] 2> 176652 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[15 (-1437949368261935104)]} 0 1
[junit4:junit4] 2> 176655 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10015 (-1437949368265080832)]} 0 1
[junit4:junit4] 2> 176658 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[18 (1437949368266129408)]} 0 3
[junit4:junit4] 2> 176660 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10018 (1437949368269275136)]} 0 2
[junit4:junit4] 2> 176663 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[18 (1437949368272420864)]} 0 2
[junit4:junit4] 2> 176667 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10018 (1437949368275566592)]} 0 3
[junit4:junit4] 2> 176694 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 176695 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 176724 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[19 (1437949368335335424)]} 0 3
[junit4:junit4] 2> 176729 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10019 (1437949368339529728)]} 0 3
[junit4:junit4] 2> 176729 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[19 (1437949368341626880)]} 0 2
[junit4:junit4] 2> 176733 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176733 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176733 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176733 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176735 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10019 (1437949368346869760)]} 0 3
[junit4:junit4] 2> 176789 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[17 (-1437949368405590016)]} 0 1
[junit4:junit4] 2> 176796 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10017 (-1437949368411881472)]} 0 1
[junit4:junit4] 2> C109_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:30949_rhm%2Fw, base_url=http://127.0.0.1:30949/rhm/w}
[junit4:junit4] 2> 176806 T285 C109 P30949 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 176807 T285 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 176809 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[17 (-1437949368409784320)]} 0 5
[junit4:junit4] 2> 176810 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10017 (-1437949368416075776)]} 0 1
[junit4:junit4] 2> 176810 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[17 (-1437949368409784320)]} 0 18
[junit4:junit4] 2> 176811 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10017 (-1437949368416075776)]} 0 12
[junit4:junit4] 2> 176816 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[20 (1437949368432852992)]} 0 2
[junit4:junit4] 2> 176818 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10020 (1437949368432852993)]} 0 4
[junit4:junit4] 2> 176827 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1437949368438095872)]} 0 3
[junit4:junit4] 2> 176828 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[20 (1437949368438095872)]} 0 9
[junit4:junit4] 2> 176829 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1437949368440193024)]} 0 4
[junit4:junit4] 2> 176829 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10020 (1437949368440193024)]} 0 8
[junit4:junit4] 2> 176924 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[21 (1437949368546099200)]} 0 2
[junit4:junit4] 2> 176925 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10021 (1437949368547147776)]} 0 2
[junit4:junit4] 2> 176933 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1437949368551342080)]} 0 2
[junit4:junit4] 2> 176934 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[21 (1437949368551342080)]} 0 7
[junit4:junit4] 2> 176935 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1437949368552390656)]} 0 2
[junit4:junit4] 2> 176936 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10021 (1437949368552390656)]} 0 8
[junit4:junit4] 2> 176976 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[22 (1437949368600625152)]} 0 2
[junit4:junit4] 2> 176978 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10022 (1437949368602722304)]} 0 2
[junit4:junit4] 2> 176987 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1437949368606916608)]} 0 2
[junit4:junit4] 2> 176988 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[22 (1437949368606916608)]} 0 8
[junit4:junit4] 2> 176989 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1437949368607965184)]} 0 3
[junit4:junit4] 2> 176990 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10022 (1437949368607965184)]} 0 9
[junit4:junit4] 2> 177074 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[23 (1437949368702337024)]} 0 2
[junit4:junit4] 2> 177075 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10023 (1437949368704434176)]} 0 2
[junit4:junit4] 2> 177084 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1437949368708628480)]} 0 2
[junit4:junit4] 2> 177085 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[23 (1437949368708628480)]} 0 8
[junit4:junit4] 2> 177086 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1437949368710725632)]} 0 2
[junit4:junit4] 2> 177087 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10023 (1437949368710725632)]} 0 8
[junit4:junit4] 2> 177149 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[24 (1437949368780980224)]} 0 3
[junit4:junit4] 2> 177150 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10024 (1437949368783077376)]} 0 2
[junit4:junit4] 2> 177159 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1437949368787271680)]} 0 2
[junit4:junit4] 2> 177160 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[24 (1437949368787271680)]} 0 8
[junit4:junit4] 2> 177160 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1437949368788320256)]} 0 2
[junit4:junit4] 2> 177161 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10024 (1437949368788320256)]} 0 8
[junit4:junit4] 2> 177232 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[22 (-1437949368870109184)]} 0 1
[junit4:junit4] 2> 177233 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10022 (-1437949368871157760)]} 0 0
[junit4:junit4] 2> 177239 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1437949368874303488)]} 0 1
[junit4:junit4] 2> 177240 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[22 (-1437949368874303488)]} 0 5
[junit4:junit4] 2> 177240 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1437949368875352064)]} 0 1
[junit4:junit4] 2> 177241 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10022 (-1437949368875352064)]} 0 5
[junit4:junit4] 2> 177246 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[25 (1437949368882692096)]} 0 3
[junit4:junit4] 2> 177247 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10025 (1437949368883740672)]} 0 3
[junit4:junit4] 2> 177256 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1437949368888983552)]} 0 2
[junit4:junit4] 2> 177257 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[25 (1437949368888983552)]} 0 8
[junit4:junit4] 2> 177257 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1437949368890032128)]} 0 2
[junit4:junit4] 2> 177259 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10025 (1437949368890032128)]} 0 9
[junit4:junit4] 2> 177268 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[26 (1437949368906809344)]} 0 2
[junit4:junit4] 2> 177269 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10026 (1437949368907857920)]} 0 2
[junit4:junit4] 2> 177279 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1437949368913100800)]} 0 2
[junit4:junit4] 2> 177280 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[26 (1437949368913100800)]} 0 8
[junit4:junit4] 2> 177280 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1437949368913100801)]} 0 2
[junit4:junit4] 2> 177281 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10026 (1437949368913100801)]} 0 9
[junit4:junit4] 2> 177383 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[23 (-1437949369028444160)]} 0 1
[junit4:junit4] 2> 177384 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10023 (-1437949369029492736)]} 0 0
[junit4:junit4] 2> 177390 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[23 (-1437949369032638464)]} 0 1
[junit4:junit4] 2> 177391 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[23 (-1437949369032638464)]} 0 5
[junit4:junit4] 2> 177391 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10023 (-1437949369033687040)]} 0 1
[junit4:junit4] 2> 177392 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10023 (-1437949369033687040)]} 0 5
[junit4:junit4] 2> 177396 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[27 (1437949369041027072)]} 0 2
[junit4:junit4] 2> 177397 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10027 (1437949369042075648)]} 0 2
[junit4:junit4] 2> 177405 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1437949369046269952)]} 0 2
[junit4:junit4] 2> 177406 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[27 (1437949369046269952)]} 0 7
[junit4:junit4] 2> 177406 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1437949369047318528)]} 0 2
[junit4:junit4] 2> 177407 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10027 (1437949369047318528)]} 0 7
[junit4:junit4] 2> 177482 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[24 (-1437949369132253184)]} 0 1
[junit4:junit4] 2> 177483 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10024 (-1437949369133301760)]} 0 1
[junit4:junit4] 2> 177489 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[24 (-1437949369135398912)]} 0 1
[junit4:junit4] 2> 177490 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[24 (-1437949369135398912)]} 0 5
[junit4:junit4] 2> 177490 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10024 (-1437949369136447488)]} 0 1
[junit4:junit4] 2> 177491 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10024 (-1437949369136447488)]} 0 5
[junit4:junit4] 2> 177495 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[28 (1437949369144836096)]} 0 2
[junit4:junit4] 2> 177496 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10028 (1437949369145884672)]} 0 2
[junit4:junit4] 2> 177504 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1437949369150078976)]} 0 2
[junit4:junit4] 2> 177505 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[28 (1437949369150078976)]} 0 7
[junit4:junit4] 2> 177505 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1437949369151127552)]} 0 2
[junit4:junit4] 2> 177506 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10028 (1437949369151127552)]} 0 7
[junit4:junit4] 2> 177522 T269 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 177522 T237 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 30949
[junit4:junit4] 2> 177522 T269 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:30949_rhm%252Fw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 177523 T237 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=262392533
[junit4:junit4] 2> 177532 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[29 (1437949369183633408)]} 0 2
[junit4:junit4] 2> 177533 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10029 (1437949369184681984)]} 0 2
[junit4:junit4] 2> 177541 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1437949369188876288)]} 0 2
[junit4:junit4] 2> 177543 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1437949369189924864)]} 0 3
[junit4:junit4] 2> 177544 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10029 (1437949369189924864)]} 0 8
[junit4:junit4] 2> 177545 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[29 (1437949369188876288)]} 0 10
[junit4:junit4] 2> 177610 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10025 (-1437949369265422336)]} 0 0
[junit4:junit4] 2> 177610 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[25 (-1437949369266470912)]} 0 0
[junit4:junit4] 2> 177618 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10025 (-1437949369271713792)]} 0 0
[junit4:junit4] 2> 177618 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[25 (-1437949369270665216)]} 0 0
[junit4:junit4] 2> 177619 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10025 (-1437949369271713792)]} 0 5
[junit4:junit4] 2> 177620 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[25 (-1437949369270665216)]} 0 7
[junit4:junit4] 2> 177624 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10030 (1437949369280102400)]} 0 2
[junit4:junit4] 2> 177626 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[30 (1437949369281150976)]} 0 3
[junit4:junit4] 2> 177635 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1437949369286393856)]} 0 2
[junit4:junit4] 2> 177636 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10030 (1437949369286393856)]} 0 8
[junit4:junit4] 2> 177637 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1437949369287442432)]} 0 2
[junit4:junit4] 2> 177638 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[30 (1437949369287442432)]} 0 9
[junit4:junit4] 2> 177680 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10027 (-1437949369338822656)]} 0 1
[junit4:junit4] 2> 177681 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[27 (-1437949369340919808)]} 0 1
[junit4:junit4] 2> 177688 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[27 (-1437949369345114112)]} 0 1
[junit4:junit4] 2> 177688 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10027 (-1437949369344065536)]} 0 1
[junit4:junit4] 2> 177689 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[27 (-1437949369345114112)]} 0 5
[junit4:junit4] 2> 177689 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10027 (-1437949369344065536)]} 0 5
[junit4:junit4] 2> 177694 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[31 (1437949369353502720)]} 0 2
[junit4:junit4] 2> 177695 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10031 (1437949369354551296)]} 0 2
[junit4:junit4] 2> 177705 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1437949369359794176)]} 0 1
[junit4:junit4] 2> 177706 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[31 (1437949369359794176)]} 0 8
[junit4:junit4] 2> 177706 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1437949369360842752)]} 0 2
[junit4:junit4] 2> 177708 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10031 (1437949369360842752)]} 0 9
[junit4:junit4] 2> 177812 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[32 (1437949369476186112)]} 0 2
[junit4:junit4] 2> 177814 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10032 (1437949369478283264)]} 0 3
[junit4:junit4] 2> 177822 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1437949369482477568)]} 0 2
[junit4:junit4] 2> 177823 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1437949369484574720)]} 0 2
[junit4:junit4] 2> 177824 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[32 (1437949369482477568)]} 0 9
[junit4:junit4] 2> 177824 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10032 (1437949369484574720)]} 0 7
[junit4:junit4] 2> 177856 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10033 (1437949369522323456)]} 0 3
[junit4:junit4] 2> 177857 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[33 (1437949369522323457)]} 0 4
[junit4:junit4] 2> 177866 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1437949369529663488)]} 0 2
[junit4:junit4] 2> 177867 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[33 (1437949369529663488)]} 0 7
[junit4:junit4] 2> 177867 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1437949369528614912)]} 0 3
[junit4:junit4] 2> 177868 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10033 (1437949369528614912)]} 0 9
[junit4:junit4] 2> 177931 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[34 (1437949369602015232)]} 0 2
[junit4:junit4] 2> 177932 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10034 (1437949369603063808)]} 0 2
[junit4:junit4] 2> 177940 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1437949369607258112)]} 0 2
[junit4:junit4] 2> 177941 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[34 (1437949369607258112)]} 0 7
[junit4:junit4] 2> 177941 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1437949369608306688)]} 0 2
[junit4:junit4] 2> 177942 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10034 (1437949369608306688)]} 0 7
[junit4:junit4] 2> 178011 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[35 (1437949369685901312)]} 0 2
[junit4:junit4] 2> 178012 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10035 (1437949369686949888)]} 0 2
[junit4:junit4] 2> 178020 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1437949369691144192)]} 0 2
[junit4:junit4] 2> 178021 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[35 (1437949369691144192)]} 0 7
[junit4:junit4] 2> 178021 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1437949369692192768)]} 0 2
[junit4:junit4] 2> 178022 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10035 (1437949369692192768)]} 0 7
[junit4:junit4] 2> 178061 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[36 (1437949369737281536)]} 0 2
[junit4:junit4] 2> 178062 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10036 (1437949369738330112)]} 0 3
[junit4:junit4] 2> 178070 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1437949369743572992)]} 0 2
[junit4:junit4] 2> 178071 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[36 (1437949369743572992)]} 0 7
[junit4:junit4] 2> 178071 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1437949369744621568)]} 0 2
[junit4:junit4] 2> 178072 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10036 (1437949369744621568)]} 0 7
[junit4:junit4] 2> 178133 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[33 (-1437949369813827584)]} 0 1
[junit4:junit4] 2> 178134 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10033 (-1437949369814876160)]} 0 1
[junit4:junit4] 2> 178140 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[33 (-1437949369818021888)]} 0 1
[junit4:junit4] 2> 178140 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[33 (-1437949369818021888)]} 0 4
[junit4:junit4] 2> 178141 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10033 (-1437949369819070464)]} 0 1
[junit4:junit4] 2> 178142 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10033 (-1437949369819070464)]} 0 5
[junit4:junit4] 2> 178146 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[37 (1437949369827459072)]} 0 2
[junit4:junit4] 2> 178147 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10037 (1437949369828507648)]} 0 2
[junit4:junit4] 2> 178155 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1437949369832701952)]} 0 2
[junit4:junit4] 2> 178156 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[37 (1437949369832701952)]} 0 7
[junit4:junit4] 2> 178156 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1437949369833750528)]} 0 2
[junit4:junit4] 2> 178157 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10037 (1437949369833750528)]} 0 7
[junit4:junit4] 2> 178223 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[36 (-1437949369908199424)]} 0 1
[junit4:junit4] 2> 178224 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10036 (-1437949369909248000)]} 0 1
[junit4:junit4] 2> 178230 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[36 (-1437949369912393728)]} 0 1
[junit4:junit4] 2> 178231 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[36 (-1437949369912393728)]} 0 4
[junit4:junit4] 2> 178231 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10036 (-1437949369914490880)]} 0 1
[junit4:junit4] 2> 178232 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10036 (-1437949369914490880)]} 0 5
[junit4:junit4] 2> 178236 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[38 (1437949369921830912)]} 0 2
[junit4:junit4] 2> 178237 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10038 (1437949369922879488)]} 0 2
[junit4:junit4] 2> 178239 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 178240 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 178245 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1437949369927073792)]} 0 2
[junit4:junit4] 2> 178246 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[38 (1437949369927073792)]} 0 7
[junit4:junit4] 2> 178246 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1437949369928122368)]} 0 2
[junit4:junit4] 2> 178247 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10038 (1437949369928122368)]} 0 7
[junit4:junit4] 2> 178282 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178282 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178282 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178282 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178326 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[39 (1437949370016202752)]} 0 2
[junit4:junit4] 2> 178327 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10039 (1437949370017251328)]} 0 2
[junit4:junit4] 2> 178331 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[39 (1437949370021445632)]} 0 2
[junit4:junit4] 2> 178332 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10039 (1437949370022494208)]} 0 2
[junit4:junit4] 2> 178435 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[40 (1437949370130497536)]} 0 2
[junit4:junit4] 2> 178436 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10040 (1437949370131546112)]} 0 2
[junit4:junit4] 2> 178440 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[40 (1437949370135740416)]} 0 2
[junit4:junit4] 2> 178441 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10040 (1437949370136788992)]} 0 2
[junit4:junit4] 2> 178487 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[41 (1437949370185023488)]} 0 2
[junit4:junit4] 2> 178488 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10041 (1437949370185023489)]} 0 2
[junit4:junit4] 2> 178493 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[41 (1437949370191314944)]} 0 2
[junit4:junit4] 2> 178494 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10041 (1437949370192363520)]} 0 2
[junit4:junit4] 2> 178585 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[42 (1437949370286735360)]} 0 2
[junit4:junit4] 2> 178586 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10042 (1437949370287783936)]} 0 2
[junit4:junit4] 2> 178590 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[42 (1437949370293026816)]} 0 2
[junit4:junit4] 2> 178591 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10042 (1437949370294075392)]} 0 2
[junit4:junit4] 2> 178659 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10041 (-1437949370366427137)]} 0 1
[junit4:junit4] 2> 178659 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[41 (-1437949370366427136)]} 0 1
[junit4:junit4] 2> 178663 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10041 (-1437949370369572864)]} 0 1
[junit4:junit4] 2> 178663 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[41 (-1437949370370621440)]} 0 1
[junit4:junit4] 2> 178668 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10043 (1437949370374815744)]} 0 2
[junit4:junit4] 2> 178669 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[43 (1437949370374815745)]} 0 3
[junit4:junit4] 2> 178673 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10043 (1437949370380058624)]} 0 2
[junit4:junit4] 2> 178674 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[43 (1437949370381107200)]} 0 2
[junit4:junit4] 2> 178764 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10044 (1437949370475479040)]} 0 1
[junit4:junit4] 2> 178765 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[44 (1437949370476527616)]} 0 2
[junit4:junit4] 2> 178770 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10044 (1437949370481770496)]} 0 2
[junit4:junit4] 2> 178771 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[44 (1437949370481770497)]} 0 3
[junit4:junit4] 2> 178865 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10045 (1437949370580336640)]} 0 2
[junit4:junit4] 2> 178866 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[45 (1437949370581385216)]} 0 3
[junit4:junit4] 2> 178870 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10045 (1437949370586628096)]} 0 2
[junit4:junit4] 2> 178871 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[45 (1437949370587676672)]} 0 2
[junit4:junit4] 2> 178874 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10042 (-1437949370590822400)]} 0 1
[junit4:junit4] 2> 178875 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[42 (-1437949370591870976)]} 0 1
[junit4:junit4] 2> 178878 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10042 (-1437949370595016704)]} 0 1
[junit4:junit4] 2> 178879 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[42 (-1437949370596065280)]} 0 1
[junit4:junit4] 2> 178883 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10046 (1437949370600259584)]} 0 2
[junit4:junit4] 2> 178884 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[46 (1437949370601308160)]} 0 2
[junit4:junit4] 2> 178888 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10046 (1437949370605502464)]} 0 2
[junit4:junit4] 2> 178889 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[46 (1437949370606551040)]} 0 2
[junit4:junit4] 2> 178954 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10045 (-1437949370675757056)]} 0 1
[junit4:junit4] 2> 178955 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[45 (-1437949370675757057)]} 0 1
[junit4:junit4] 2> 178958 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10045 (-1437949370678902784)]} 0 1
[junit4:junit4] 2> 178959 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[45 (-1437949370679951360)]} 0 1
[junit4:junit4] 2> 178963 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10047 (1437949370684145664)]} 0 2
[junit4:junit4] 2> 178964 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[47 (1437949370685194240)]} 0 2
[junit4:junit4] 2> 178968 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10047 (1437949370689388544)]} 0 2
[junit4:junit4] 2> 178969 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[47 (1437949370690437120)]} 0 2
[junit4:junit4] 2> 179018 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10046 (-1437949370742865920)]} 0 0
[junit4:junit4] 2> 179019 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[46 (-1437949370742865921)]} 0 1
[junit4:junit4] 2> 179022 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10046 (-1437949370746011648)]} 0 1
[junit4:junit4] 2> 179023 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[46 (-1437949370747060224)]} 0 1
[junit4:junit4] 2> 179027 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10048 (1437949370751254528)]} 0 2
[junit4:junit4] 2> 179028 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[48 (1437949370752303104)]} 0 2
[junit4:junit4] 2> 179032 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10048 (1437949370756497408)]} 0 2
[junit4:junit4] 2> 179033 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[48 (1437949370757545984)]} 0 2
[junit4:junit4] 2> 179134 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10049 (1437949370862403584)]} 0 2
[junit4:junit4] 2> 179135 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[49 (1437949370862403585)]} 0 3
[junit4:junit4] 2> 179139 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10049 (1437949370868695040)]} 0 2
[junit4:junit4] 2> 179140 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[49 (1437949370869743616)]} 0 2
[junit4:junit4] 2> 179157 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10050 (1437949370887569408)]} 0 2
[junit4:junit4] 2> 179158 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[50 (1437949370888617984)]} 0 2
[junit4:junit4] 2> 179162 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10050 (1437949370892812288)]} 0 2
[junit4:junit4] 2> 179163 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[50 (1437949370893860864)]} 0 2
[junit4:junit4] 2> 179255 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10047 (-1437949370991378432)]} 0 1
[junit4:junit4] 2> 179259 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10047 (-1437949370995572736)]} 0 1
[junit4:junit4] 2> 179264 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10051 (1437949370999767040)]} 0 2
[junit4:junit4] 2> 179269 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10051 (1437949371005009920)]} 0 2
[junit4:junit4] 2> 179270 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[47 (-1437949371006058496)]} 0 1
[junit4:junit4] 2> 179274 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[47 (-1437949371010252800)]} 0 1
[junit4:junit4] 2> 179279 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[51 (1437949371015495680)]} 0 2
[junit4:junit4] 2> 179284 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[51 (1437949371020738560)]} 0 2
[junit4:junit4] 2> 179324 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10048 (-1437949371063730176)]} 0 0
[junit4:junit4] 2> 179329 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10048 (-1437949371067924480)]} 0 1
[junit4:junit4] 2> 179334 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10052 (1437949371073167360)]} 0 2
[junit4:junit4] 2> 179339 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10052 (1437949371078410240)]} 0 2
[junit4:junit4] 2> 179340 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[48 (-1437949371080507392)]} 0 1
[junit4:junit4] 2> 179344 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[48 (-1437949371083653120)]} 0 1
[junit4:junit4] 2> 179349 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[52 (1437949371088896000)]} 0 2
[junit4:junit4] 2> 179354 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[52 (1437949371094138880)]} 0 2
[junit4:junit4] 2> 179390 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10049 (-1437949371132936192)]} 0 1
[junit4:junit4] 2> 179394 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10049 (-1437949371136081920)]} 0 1
[junit4:junit4] 2> 179399 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10053 (1437949371141324800)]} 0 2
[junit4:junit4] 2> 179404 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10053 (1437949371146567680)]} 0 2
[junit4:junit4] 2> 179405 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[49 (-1437949371147616256)]} 0 1
[junit4:junit4] 2> 179409 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[49 (-1437949371151810560)]} 0 1
[junit4:junit4] 2> 179414 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[53 (1437949371157053440)]} 0 2
[junit4:junit4] 2> 179417 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10054 (1437949371160199168)]} 0 2
[junit4:junit4] 2> 179419 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[53 (1437949371162296320)]} 0 2
[junit4:junit4] 2> 179422 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10054 (1437949371165442048)]} 0 2
[junit4:junit4] 2> 179433 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[54 (1437949371176976384)]} 0 2
[junit4:junit4] 2> 179437 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10050 (-1437949371182219264)]} 0 1
[junit4:junit4] 2> 179438 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[54 (1437949371182219264)]} 0 2
[junit4:junit4] 2> 179441 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10050 (-1437949371186413568)]} 0 1
[junit4:junit4] 2> 179446 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10055 (1437949371190607872)]} 0 2
[junit4:junit4] 2> 179452 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10055 (1437949371195850752)]} 0 2
[junit4:junit4] 2> 179453 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[50 (-1437949371198996480)]} 0 0
[junit4:junit4] 2> 179457 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[50 (-1437949371203190784)]} 0 1
[junit4:junit4] 2> 179462 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[55 (1437949371207385088)]} 0 2
[junit4:junit4] 2> 179467 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[55 (1437949371212627968)]} 0 2
[junit4:junit4] 2> C109_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:30949_rhm%2Fw, base_url=http://127.0.0.1:30949/rhm/w}
[junit4:junit4] 2> 179524 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:30946/rhm/w/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 179524 T298 C109 P30949 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:30949/rhm/w START replicas=[http://127.0.0.1:30946/rhm/w/collection1/] nUpdates=100
[junit4:junit4] 2> 179525 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10056 (1437949371273445376)]} 0 2
[junit4:junit4] 2> 179527 T237 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 179527 T237 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 179530 T237 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=3core=collection1
[junit4:junit4] 2> 179531 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10056 (1437949371279736832)]} 0 2
[junit4:junit4] 2> 179532 T272 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 179531 T298 C109 P30949 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 179532 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 179533 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 179533 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 179533 T298 C109 P30949 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 179534 T298 C109 P30949 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:30946/rhm/w/collection1/. core=collection1
[junit4:junit4] 2> 179534 T298 C109 P30949 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 179539 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[56 (1437949371288125440)]} 0 2
[junit4:junit4] 2> 179544 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[56 (1437949371293368320)]} 0 2
[junit4:junit4] 2> 179547 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10052 (-1437949371297562624)]} 0 1
[junit4:junit4] 2> 179549 T273 C111 P30946 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 179552 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10052 (-1437949371300708352)]} 0 2
[junit4:junit4] 2> 179557 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10057 (1437949371306999808)]} 0 2
[junit4:junit4] 2> 179559 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[52 (-1437949371309096960)]} 0 1
[junit4:junit4] 2> 179562 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10057 (1437949371312242688)]} 0 2
[junit4:junit4] 2> 179563 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[52 (-1437949371313291264)]} 0 1
[junit4:junit4] 2> 179568 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[57 (1437949371318534144)]} 0 2
[junit4:junit4] 2> 179573 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[57 (1437949371323777024)]} 0 2
[junit4:junit4] 2> 179618 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10058 (1437949371370962944)]} 0 2
[junit4:junit4] 2> 179624 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10058 (1437949371376205824)]} 0 2
[junit4:junit4] 2> 179628 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[58 (1437949371381448704)]} 0 2
[junit4:junit4] 2> 179633 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[58 (1437949371386691584)]} 0 2
[junit4:junit4] 2> 179650 T273 C111 P30946 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 179651 T273 C111 P30946 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 179713 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10054 (-1437949371471626240)]} 0 0
[junit4:junit4] 2> 179718 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10054 (-1437949371476869120)]} 0 1
[junit4:junit4] 2> 179720 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[54 (-1437949371477917696)]} 0 1
[junit4:junit4] 2> 179723 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10059 (1437949371481063424)]} 0 2
[junit4:junit4] 2> 179724 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[54 (-1437949371482112000)]} 0 1
[junit4:junit4] 2> 179728 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10059 (1437949371486306304)]} 0 2
[junit4:junit4] 2> 179729 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[59 (1437949371487354880)]} 0 2
[junit4:junit4] 2> 179734 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[59 (1437949371492597760)]} 0 2
[junit4:junit4] 2> 179751 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10055 (-1437949371511472128)]} 0 1
[junit4:junit4] 2> 179755 T273 C111 P30946 oass.SolrIndexSearcher.<init> Opening Searcher@2b6eec1 realtime
[junit4:junit4] 2> 179755 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10055 (-1437949371515666432)]} 0 0
[junit4:junit4] 2> 179756 T273 C111 P30946 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 179756 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[55 (-1437949371515666432)]} 0 1
[junit4:junit4] 2> 179756 T273 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 208
[junit4:junit4] 2> 179758 T298 C109 P30949 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 179758 T298 C109 P30949 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 179760 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[55 (-1437949371519860736)]} 0 1
[junit4:junit4] 2> 179760 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10060 (1437949371519860736)]} 0 2
[junit4:junit4] 2> 179761 T274 C111 P30946 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 179762 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 179763 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 179763 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 179763 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 179765 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[60 (1437949371525103616)]} 0 2
[junit4:junit4] 2> 179766 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10060 (1437949371526152192)]} 0 2
[junit4:junit4] 2> 179766 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 179767 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 179768 T298 C109 P30949 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920
[junit4:junit4] 2> 179769 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@44c0fc36 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f9777af) fullCopy=false
[junit4:junit4] 2> 179770 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[60 (1437949371530346496)]} 0 2
[junit4:junit4] 2> 179771 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179776 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179781 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179785 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179788 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 179789 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 179790 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 179791 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 179793 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 179793 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 179794 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 179800 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10056 (-1437949371562852352)]} 0 0
[junit4:junit4] 2> 179802 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179802 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179802 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179802 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179803 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[56 (-1437949371565998080)]} 0 1
[junit4:junit4] 2> 179805 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10056 (-1437949371568095232)]} 0 1
[junit4:junit4] 2> 179808 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[56 (-1437949371571240960)]} 0 1
[junit4:junit4] 2> 179810 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10061 (1437949371572289536)]} 0 2
[junit4:junit4] 2> 179813 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[61 (1437949371575435264)]} 0 2
[junit4:junit4] 2> 179815 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10061 (1437949371577532416)]} 0 2
[junit4:junit4] 2> 179818 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[61 (1437949371580678144)]} 0 2
[junit4:junit4] 2> 179818 T298 C109 P30949 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 179819 T298 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 179823 T298 C109 P30949 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 179823 T298 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 179824 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 179827 T298 C109 P30949 oass.SolrIndexSearcher.<init> Opening Searcher@7142666b main
[junit4:junit4] 2> 179828 T297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7142666b main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c38/14)}
[junit4:junit4] 2> 179829 T298 C109 P30949 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920;done=true>>]
[junit4:junit4] 2> 179829 T298 C109 P30949 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920
[junit4:junit4] 2> 179830 T298 C109 P30949 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920
[junit4:junit4] 2> 179838 T298 C109 P30949 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 179838 T306 C109 P30949 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=5468
[junit4:junit4] 2> 179845 T306 C109 P30949 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 179847 T306 C109 P30949 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_3,generation=3}
[junit4:junit4] 2> 179847 T306 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 179848 T306 C109 P30949 oass.SolrIndexSearcher.<init> Opening Searcher@33f376d2 main
[junit4:junit4] 2> 179848 T306 C109 P30949 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 179849 T297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33f376d2 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c38/14)}
[junit4:junit4] 2> 179850 T306 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] {} 0 12
[junit4:junit4] 2> 179850 T306 C109 P30949 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=5468}
[junit4:junit4] 2> 179851 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 179851 T298 C109 P30949 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 179851 T298 C109 P30949 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 179854 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 179855 T237 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@a41ba9d
[junit4:junit4] 2> 179864 T237 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=38,cumulative_deletesById=16,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 179865 T237 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 179866 T237 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 179866 T237 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 179869 T237 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 179870 T237 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 179870 T237 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2;done=false>>]
[junit4:junit4] 2> 179871 T237 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2
[junit4:junit4] 2> 179871 T237 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index;done=false>>]
[junit4:junit4] 2> 179872 T237 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index
[junit4:junit4] 2> 179872 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10057 (-1437949371638349824)]} 0 0
[junit4:junit4] 2> 179874 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179875 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[57 (-1437949371640446976)]} 0 1
[junit4:junit4] 2> 179876 T259 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 179876 T279 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 179875 T237 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/rhm/w,null}
[junit4:junit4] 2> 179877 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10057 (-1437949371643592704)]} 0 1
[junit4:junit4] 2> 179876 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 179876 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 179877 T266 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 179880 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[57 (-1437949371646738432)]} 0 1
[junit4:junit4] 2> 179882 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10062 (1437949371647787008)]} 0 2
[junit4:junit4] 2> 179885 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[62 (1437949371650932736)]} 0 2
[junit4:junit4] 2> 179887 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10062 (1437949371653029888)]} 0 2
[junit4:junit4] 2> 179890 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[62 (1437949371656175616)]} 0 2
[junit4:junit4] 2> 179909 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10063 (1437949371676098560)]} 0 2
[junit4:junit4] 2> 179911 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[63 (1437949371678195712)]} 0 2
[junit4:junit4] 2> 179915 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10063 (1437949371682390016)]} 0 2
[junit4:junit4] 2> 179916 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[63 (1437949371683438592)]} 0 2
[junit4:junit4] 2> 179948 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10058 (-1437949371718041600)]} 0 1
[junit4:junit4] 2> 179950 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[58 (-1437949371719090176)]} 0 1
[junit4:junit4] 2> 179952 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10058 (-1437949371721187328)]} 0 1
[junit4:junit4] 2> 179954 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[58 (-1437949371724333056)]} 0 1
[junit4:junit4] 2> 179957 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10064 (1437949371726430208)]} 0 2
[junit4:junit4] 2> 179962 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[64 (1437949371728527360)]} 0 5
[junit4:junit4] 2> 179962 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10064 (1437949371730624512)]} 0 2
[junit4:junit4] 2> 179967 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[64 (1437949371736915968)]} 0 2
[junit4:junit4] 2> 180026 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10065 (1437949371798781952)]} 0 1
[junit4:junit4] 2> 180030 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[65 (1437949371802976256)]} 0 2
[junit4:junit4] 2> 180032 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10065 (1437949371805073408)]} 0 2
[junit4:junit4] 2> 180035 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[65 (1437949371808219136)]} 0 2
[junit4:junit4] 2> 180099 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10066 (1437949371875328000)]} 0 1
[junit4:junit4] 2> 180101 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[66 (1437949371877425152)]} 0 2
[junit4:junit4] 2> 180105 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10066 (1437949371881619456)]} 0 2
[junit4:junit4] 2> 180106 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[66 (1437949371882668032)]} 0 2
[junit4:junit4] 2> 180132 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10067 (1437949371909931008)]} 0 2
[junit4:junit4] 2> 180133 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[67 (1437949371910979584)]} 0 2
[junit4:junit4] 2> 180137 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10067 (1437949371915173888)]} 0 2
[junit4:junit4] 2> 180138 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[67 (1437949371916222464)]} 0 2
[junit4:junit4] 2> 180217 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10059 (-1437949372000108544)]} 0 1
[junit4:junit4] 2> 180219 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[59 (-1437949372001157120)]} 0 1
[junit4:junit4] 2> 180221 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10059 (-1437949372003254272)]} 0 1
[junit4:junit4] 2> 180223 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[59 (-1437949372006400000)]} 0 1
[junit4:junit4] 2> 180226 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10068 (1437949372008497152)]} 0 2
[junit4:junit4] 2> 180228 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[68 (1437949372010594304)]} 0 2
[junit4:junit4] 2> 180231 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10068 (1437949372013740032)]} 0 2
[junit4:junit4] 2> 180233 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[68 (1437949372015837184)]} 0 2
[junit4:junit4] 2> 180301 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10069 (1437949372087140352)]} 0 1
[junit4:junit4] 2> 180303 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[69 (1437949372089237504)]} 0 2
[junit4:junit4] 2> 180306 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10069 (1437949372092383232)]} 0 2
[junit4:junit4] 2> 180308 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[69 (1437949372094480384)]} 0 2
[junit4:junit4] 2> 180310 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10060 (-1437949372096577536)]} 0 1
[junit4:junit4] 2> 180312 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[60 (-1437949372098674688)]} 0 1
[junit4:junit4] 2> 180314 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10060 (-1437949372101820416)]} 0 1
[junit4:junit4] 2> 180316 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[60 (-1437949372102868992)]} 0 1
[junit4:junit4] 2> 180319 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10070 (1437949372104966144)]} 0 2
[junit4:junit4] 2> 180321 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[70 (1437949372108111872)]} 0 2
[junit4:junit4] 2> 180324 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10070 (1437949372111257600)]} 0 2
[junit4:junit4] 2> 180326 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[70 (1437949372113354752)]} 0 2
[junit4:junit4] 2> 180337 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10062 (-1437949372124889088)]} 0 1
[junit4:junit4] 2> 180339 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[62 (-1437949372126986240)]} 0 0
[junit4:junit4] 2> 180341 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10062 (-1437949372130131968)]} 0 1
[junit4:junit4] 2> 180342 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[62 (-1437949372130131969)]} 0 1
[junit4:junit4] 2> 180346 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10071 (1437949372134326272)]} 0 2
[junit4:junit4] 2> 180347 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[71 (1437949372135374848)]} 0 2
[junit4:junit4] 2> 180351 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10071 (1437949372139569152)]} 0 2
[junit4:junit4] 2> 180352 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[71 (1437949372140617728)]} 0 2
[junit4:junit4] 2> 180424 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10072 (1437949372216115200)]} 0 2
[junit4:junit4] 2> 180424 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[72 (1437949372216115201)]} 0 1
[junit4:junit4] 2> 180429 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10072 (1437949372221358080)]} 0 2
[junit4:junit4] 2> 180430 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[72 (1437949372222406656)]} 0 2
[junit4:junit4] 2> 180483 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10073 (1437949372277981184)]} 0 2
[junit4:junit4] 2> 180484 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[73 (1437949372279029760)]} 0 2
[junit4:junit4] 2> 180488 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10073 (1437949372283224064)]} 0 2
[junit4:junit4] 2> 180489 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[73 (1437949372284272640)]} 0 2
[junit4:junit4] 2> 180550 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10063 (-1437949372349284352)]} 0 1
[junit4:junit4] 2> 180551 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[63 (-1437949372350332928)]} 0 1
[junit4:junit4] 2> 180553 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10063 (-1437949372352430080)]} 0 0
[junit4:junit4] 2> 180554 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[63 (-1437949372353478656)]} 0 0
[junit4:junit4] 2> 180557 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10074 (1437949372355575808)]} 0 1
[junit4:junit4] 2> 180558 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[74 (1437949372356624384)]} 0 1
[junit4:junit4] 2> 180561 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10074 (1437949372359770112)]} 0 1
[junit4:junit4] 2> 180562 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[74 (1437949372360818688)]} 0 1
[junit4:junit4] 2> 180582 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10075 (1437949372381790208)]} 0 1
[junit4:junit4] 2> 180584 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[75 (1437949372382838784)]} 0 2
[junit4:junit4] 2> 180586 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10075 (1437949372385984512)]} 0 1
[junit4:junit4] 2> 180589 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[75 (1437949372389130240)]} 0 2
[junit4:junit4] 2> 180608 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10064 (-1437949372409053184)]} 0 1
[junit4:junit4] 2> 180610 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[64 (-1437949372411150336)]} 0 1
[junit4:junit4] 2> 180612 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10064 (-1437949372413247488)]} 0 1
[junit4:junit4] 2> 180613 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[64 (-1437949372415344640)]} 0 0
[junit4:junit4] 2> 180616 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10076 (1437949372416393216)]} 0 2
[junit4:junit4] 2> 180617 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[76 (1437949372418490368)]} 0 1
[junit4:junit4] 2> 180619 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10076 (1437949372420587520)]} 0 1
[junit4:junit4] 2> 180621 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[76 (1437949372422684672)]} 0 1
[junit4:junit4] 2> 180703 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10077 (1437949372508667904)]} 0 2
[junit4:junit4] 2> 180705 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[77 (1437949372510765
[...truncated too long message...]
loseSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1839970 T3763 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1839970 T3763 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4/index;done=false>>]
[junit4:junit4] 2> 1839971 T3763 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4/index
[junit4:junit4] 2> 1839971 T3763 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4;done=false>>]
[junit4:junit4] 2> 1839971 T3763 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4
[junit4:junit4] 2> 1839972 T3763 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 1839972 T3763 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election/89871929881460748-5-n_0000000001
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:260)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1839973 T3763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1839973 T3763 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1839973 T3763 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1839974 T3763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1839974 T3763 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1839974 T3763 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=uk, timezone=Pacific/Yap
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=255022080,total=491782144
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestOmitPositions, TestDefaultSimilarityFactory, IndexReaderFactoryTest, BasicDistributedZk2Test, ClusterStateTest, HighlighterTest, TestSystemIdResolver, TestStressLucene, TestZkChroot, TestJmxIntegration, RequiredFieldsTest, TestIndexingPerformance, SortByFunctionTest, DistanceFunctionTest, UniqFieldsUpdateProcessorFactoryTest, TestCopyFieldCollectionResource, FieldAnalysisRequestHandlerTest, BasicZkTest, TestCSVResponseWriter, TestDistributedGrouping, TestBinaryField, TestHashPartitioner, DefaultValueUpdateProcessorTest, TestUpdate, TestArbitraryIndexDir, TestSchemaNameResource, TestRandomFaceting, TestReload, TestRecovery, SampleTest, TestSuggestSpellingConverter, DirectSolrSpellCheckerTest, TestFieldResource, TestManagedSchemaFieldResource, TestComponentsName, SignatureUpdateProcessorFactoryTest, TestSolrQueryParserResource, JSONWriterTest, TestCoreDiscovery, TestPostingsSolrHighlighter, IndexSchemaRuntimeFieldTest, TestLuceneMatchVersion, TestClassNameShortening, TestAtomicUpdateErrorCases, TestFastOutputStream, MultiTermTest, TestUniqueKeyFieldResource, ShardRoutingCustomTest, FileUtilsTest, SolrCmdDistributorTest, TestSolr4Spatial, TestShardHandlerFactory, PingRequestHandlerTest, SolrPluginUtilsTest, StandardRequestHandlerTest, TestCSVLoader, TestDefaultSearchFieldResource, TestQueryTypes, TestNumberUtils, TestDynamicFieldCollectionResource, SpellCheckCollatorTest, TestSolrXMLSerializer, TestPartialUpdateDeduplication, TestLMJelinekMercerSimilarityFactory, CurrencyFieldOpenExchangeTest, TestFiltering, NotRequiredUniqueKeyTest, ChaosMonkeySafeLeaderTest, TestIndexSearcher, TestRTGBase, SuggesterWFSTTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, ZkCLITest, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestStressReorder, HardAutoCommitTest, TestRangeQuery, SolrCoreTest, StatsComponentTest, TestGroupingSearch, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SolrRequestParserTest, DocValuesTest, SuggesterTest, NoCacheHeaderTest, TestPseudoReturnFields, TestRemoteStreaming, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, TestCollationField, QueryParsingTest, JsonLoaderTest, CSVRequestHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, TestPropInject, UpdateParamsTest, TestSolrIndexConfig, CopyFieldTest, SolrIndexConfigTest, NumericFieldsTest, TestElisionMultitermQuery, TestPhraseSuggestions, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, DOMUtilTest, TestSolrJ, TestUtils, SliceStateTest, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed on J1 in 223.70s, 1 test, 1 error <<< FAILURES!
[...truncated 54 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:386: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:366: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:887: There were test failures: 296 suites, 1243 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 52 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4068 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4068/
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:20986 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:20986 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([7262B9B042D2C205:F38437A8358DA239]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 482 from http://127.0.0.1:41104/collection1lastClient and got 476 from http://127.0.0.1:33657/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 482 from http://127.0.0.1:41104/collection1lastClient and got 476 from http://127.0.0.1:33657/collection1
at __randomizedtesting.SeedInfo.seed([7262B9B042D2C205:F38437A8358DA239]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9542 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 946420 T1801 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 946428 T1801 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyShardSplitTest-1371299031648
[junit4:junit4] 2> 946430 T1801 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 946430 T1802 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 946531 T1801 oasc.ZkTestServer.run start zk server on port:34237
[junit4:junit4] 2> 946532 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 946536 T1808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d61e46a name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 946536 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 946537 T1801 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 946555 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 946557 T1810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14547f9a name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 946557 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 946558 T1801 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 946562 T1801 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 946573 T1801 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 946575 T1801 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 946579 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 946579 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 946592 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 946592 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 946707 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 946708 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 946719 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 946720 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 946723 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 946724 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 946727 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 946728 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 946731 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 946731 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 946734 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 946735 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 946738 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 946739 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 947020 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 947026 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11430
[junit4:junit4] 2> 947026 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 947027 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 947028 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968
[junit4:junit4] 2> 947028 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/solr.xml
[junit4:junit4] 2> 947029 T1801 oasc.CoreContainer.<init> New CoreContainer 1424141568
[junit4:junit4] 2> 947029 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/'
[junit4:junit4] 2> 947030 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/'
[junit4:junit4] 2> 947160 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 947161 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 947162 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 947162 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 947163 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 947163 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 947164 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 947164 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 947165 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 947166 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 947183 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 947184 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 947184 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 947185 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 947213 T1821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43576488 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 947214 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 947220 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 947233 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 947280 T1823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3069cf63 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 947280 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 947295 T1801 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 947394 T1801 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 947413 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 947415 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11430_
[junit4:junit4] 2> 947461 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11430_
[junit4:junit4] 2> 947465 T1801 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 947480 T1801 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 947483 T1801 oasc.Overseer.start Overseer (id=89869453338607619-127.0.0.1:11430_-n_0000000000) starting
[junit4:junit4] 2> 947494 T1801 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 947505 T1825 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 947506 T1801 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 947509 T1801 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 947511 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 947513 T1824 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 947517 T1826 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 947517 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 949017 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 949018 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:11430_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:11430"}
[junit4:junit4] 2> 949018 T1824 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 949018 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 949031 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 949521 T1826 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1
[junit4:junit4] 2> 949521 T1826 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 949522 T1826 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 949522 T1826 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 949524 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/'
[junit4:junit4] 2> 949525 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/lib/README' to classloader
[junit4:junit4] 2> 949526 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 949576 T1826 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 949634 T1826 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 949735 T1826 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 949742 T1826 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 950331 T1826 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 950336 T1826 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 950338 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 950343 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 950368 T1826 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 950368 T1826 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/
[junit4:junit4] 2> 950369 T1826 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 950369 T1826 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 950370 T1826 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data
[junit4:junit4] 2> 950370 T1826 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/index/
[junit4:junit4] 2> 950371 T1826 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 950371 T1826 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/index
[junit4:junit4] 2> 950376 T1826 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_1,generation=1}
[junit4:junit4] 2> 950376 T1826 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 950379 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 950380 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 950381 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 950381 T1826 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 950382 T1826 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 950382 T1826 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 950382 T1826 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 950383 T1826 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 950383 T1826 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 950400 T1826 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 950407 T1826 oass.SolrIndexSearcher.<init> Opening Searcher@710cac4c main
[junit4:junit4] 2> 950408 T1826 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/tlog
[junit4:junit4] 2> 950409 T1826 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 950409 T1826 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 950413 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@710cac4c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 950415 T1826 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 950415 T1826 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11430 collection:control_collection shard:shard1
[junit4:junit4] 2> 950416 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 950448 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 950460 T1826 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 950461 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 950461 T1826 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11430/collection1/
[junit4:junit4] 2> 950461 T1826 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 950462 T1826 oasc.SyncStrategy.syncToMe http://127.0.0.1:11430/collection1/ has no replicas
[junit4:junit4] 2> 950462 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11430/collection1/
[junit4:junit4] 2> 950462 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 950536 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 950553 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.register We are http://127.0.0.1:11430/collection1/ and leader is http://127.0.0.1:11430/collection1/
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11430
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 950577 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 950578 T1826 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 950579 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 950580 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 950581 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 950597 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 950598 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 950600 T1830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@249843d6 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 950601 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 950603 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 950605 T1801 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 950873 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 950876 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20986
[junit4:junit4] 2> 950877 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 950877 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 950878 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825
[junit4:junit4] 2> 950878 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/solr.xml
[junit4:junit4] 2> 950879 T1801 oasc.CoreContainer.<init> New CoreContainer 1196405988
[junit4:junit4] 2> 950880 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/'
[junit4:junit4] 2> 950880 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/'
[junit4:junit4] 2> 951011 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 951012 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 951012 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 951013 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 951013 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 951014 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 951014 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 951015 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 951015 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 951016 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 951032 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 951032 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 951033 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 951034 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 951037 T1841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f801b05 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 951038 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 951053 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 951064 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 951066 T1843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26df2e name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 951066 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 951072 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 952067 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 952068 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:11430_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:11430"}
[junit4:junit4] 2> 952105 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20986_
[junit4:junit4] 2> 952180 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20986_
[junit4:junit4] 2> 952206 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952206 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952206 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952262 T1843 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 952262 T1830 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 952262 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952269 T1823 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 952273 T1844 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 952273 T1844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 953764 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 953765 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20986_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20986"}
[junit4:junit4] 2> 953765 T1824 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 953765 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 953778 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 953778 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 953778 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 954277 T1844 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1
[junit4:junit4] 2> 954277 T1844 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 954278 T1844 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 954278 T1844 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 954280 T1844 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/'
[junit4:junit4] 2> 954281 T1844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/lib/README' to classloader
[junit4:junit4] 2> 954282 T1844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 954335 T1844 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 954397 T1844 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 954498 T1844 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 954505 T1844 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 955086 T1844 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 955090 T1844 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 955092 T1844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 955097 T1844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 955121 T1844 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 955122 T1844 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/
[junit4:junit4] 2> 955122 T1844 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 955122 T1844 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 955123 T1844 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1
[junit4:junit4] 2> 955123 T1844 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/index/
[junit4:junit4] 2> 955124 T1844 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 955125 T1844 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/index
[junit4:junit4] 2> 955129 T1844 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 955129 T1844 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 955132 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 955133 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 955134 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 955134 T1844 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 955135 T1844 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 955135 T1844 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 955136 T1844 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 955136 T1844 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 955137 T1844 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 955153 T1844 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 955161 T1844 oass.SolrIndexSearcher.<init> Opening Searcher@4c6923b5 main
[junit4:junit4] 2> 955161 T1844 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/tlog
[junit4:junit4] 2> 955162 T1844 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 955163 T1844 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 955167 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c6923b5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 955170 T1844 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 955170 T1844 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20986 collection:collection1 shard:shard1
[junit4:junit4] 2> 955171 T1844 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 955187 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 955190 T1844 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 955190 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 955190 T1844 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 955190 T1844 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 955191 T1844 oasc.SyncStrategy.syncToMe http://127.0.0.1:20986/collection1/ has no replicas
[junit4:junit4] 2> 955191 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 955191 T1844 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 955282 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 955299 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 955299 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 955299 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.register We are http://127.0.0.1:20986/collection1/ and leader is http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20986
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 955351 T1844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 955352 T1844 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 955353 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 955354 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 955355 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 955612 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 955615 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:62105
[junit4:junit4] 2> 955615 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 955616 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 955616 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590
[junit4:junit4] 2> 955617 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/solr.xml
[junit4:junit4] 2> 955617 T1801 oasc.CoreContainer.<init> New CoreContainer 1506313995
[junit4:junit4] 2> 955618 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/'
[junit4:junit4] 2> 955618 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/'
[junit4:junit4] 2> 955739 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 955740 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 955740 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 955741 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 955741 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 955742 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 955742 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 955743 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 955743 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 955744 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 955760 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 955761 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 955761 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 955762 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 955766 T1857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bcf40e0 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 955767 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 955769 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 955782 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 955784 T1859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50797dd7 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 955784 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 955796 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 956800 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62105_
[junit4:junit4] 2> 956802 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:62105_
[junit4:junit4] 2> 956806 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 956806 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 956806 T1830 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956806 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 956807 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956807 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20986_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20986"}
[junit4:junit4] 2> 956806 T1859 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956817 T1843 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956820 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956820 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956820 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956823 T1860 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 956823 T1860 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 956832 T1823 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956833 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958333 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 958334 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:62105_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:62105"}
[junit4:junit4] 2> 958335 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 958335 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 958348 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958348 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958348 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958348 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958833 T1860 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1
[junit4:junit4] 2> 958833 T1860 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 958834 T1860 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 958834 T1860 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 958836 T1860 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/'
[junit4:junit4] 2> 958837 T1860 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/lib/README' to classloader
[junit4:junit4] 2> 958838 T1860 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 958892 T1860 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 958955 T1860 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 959056 T1860 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 959063 T1860 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 959644 T1860 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 959648 T1860 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 959651 T1860 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 959656 T1860 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 959680 T1860 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 959680 T1860 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/
[junit4:junit4] 2> 959681 T1860 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 959681 T1860 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 959682 T1860 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2
[junit4:junit4] 2> 959682 T1860 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/index/
[junit4:junit4] 2> 959683 T1860 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 959683 T1860 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/index
[junit4:junit4] 2> 959687 T1860 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_1,generation=1}
[junit4:junit4] 2> 959687 T1860 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 959690 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 959691 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 959692 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 959692 T1860 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 959693 T1860 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 959693 T1860 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 959694 T1860 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 959694 T1860 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 959695 T1860 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 959710 T1860 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 959717 T1860 oass.SolrIndexSearcher.<init> Opening Searcher@3f73983a main
[junit4:junit4] 2> 959717 T1860 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/tlog
[junit4:junit4] 2> 959718 T1860 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 959718 T1860 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 959723 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f73983a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 959725 T1860 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 959726 T1860 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:62105 collection:collection1 shard:shard2
[junit4:junit4] 2> 959726 T1860 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 959744 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 959747 T1860 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 959747 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 959747 T1860 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 959747 T1860 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 959748 T1860 oasc.SyncStrategy.syncToMe http://127.0.0.1:62105/collection1/ has no replicas
[junit4:junit4] 2> 959748 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 959748 T1860 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 959852 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 959868 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959868 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959868 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959868 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.register We are http://127.0.0.1:62105/collection1/ and leader is http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:62105
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 959907 T1860 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 959909 T1860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 959910 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 959910 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 959911 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 960173 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 960177 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:41605
[junit4:junit4] 2> 960178 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 960178 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 960179 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144
[junit4:junit4] 2> 960179 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/solr.xml
[junit4:junit4] 2> 960180 T1801 oasc.CoreContainer.<init> New CoreContainer 578983594
[junit4:junit4] 2> 960180 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/'
[junit4:junit4] 2> 960181 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/'
[junit4:junit4] 2> 960302 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 960302 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 960303 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 960303 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 960304 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 960304 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 960305 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 960305 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 960306 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 960306 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 960320 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 960321 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 960322 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 960323 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 960326 T1873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a67455b name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 960326 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 960342 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 960353 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 960355 T1875 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d4cc7fa name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 960355 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 960360 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 961364 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41605_
[junit4:junit4] 2> 961366 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41605_
[junit4:junit4] 2> 961370 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 961370 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 961370 T1859 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961370 T1875 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961370 T1830 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961371 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961371 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961372 T1823 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961379 T1843 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961380 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 961381 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:62105_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:62105"}
[junit4:junit4] 2> 961384 T1876 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 961385 T1876 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 961394 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962900 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 962901 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41605_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41605"}
[junit4:junit4] 2> 962901 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 962901 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 962925 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 963396 T1876 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1
[junit4:junit4] 2> 963396 T1876 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 963397 T1876 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 963397 T1876 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 963399 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/'
[junit4:junit4] 2> 963400 T1876 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/lib/README' to classloader
[junit4:junit4] 2> 963401 T1876 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 963454 T1876 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 963516 T1876 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 963617 T1876 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 963624 T1876 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 964184 T1876 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 964187 T1876 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 964189 T1876 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 964193 T1876 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 964214 T1876 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 964214 T1876 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/
[junit4:junit4] 2> 964214 T1876 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 964215 T1876 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 964215 T1876 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3
[junit4:junit4] 2> 964216 T1876 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index/
[junit4:junit4] 2> 964216 T1876 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 964217 T1876 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index
[junit4:junit4] 2> 964220 T1876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 964220 T1876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 964222 T1876 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 964222 T1876 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 964223 T1876 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 964223 T1876 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 964224 T1876 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 964224 T1876 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 964224 T1876 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 964225 T1876 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 964225 T1876 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 964236 T1876 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 964242 T1876 oass.SolrIndexSearcher.<init> Opening Searcher@349da0e2 main
[junit4:junit4] 2> 964242 T1876 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/tlog
[junit4:junit4] 2> 964243 T1876 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 964243 T1876 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 964246 T1877 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@349da0e2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 964248 T1876 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 964248 T1876 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41605 collection:collection1 shard:shard1
[junit4:junit4] 2> 964252 T1876 oasc.ZkController.register We are http://127.0.0.1:41605/collection1/ and leader is http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 964252 T1876 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41605
[junit4:junit4] 2> 964252 T1876 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 964252 T1876 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C712 name=collection1 org.apache.solr.core.SolrCore@65177b81 url=http://127.0.0.1:41605/collection1 node=127.0.0.1:41605_ C712_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41605_, base_url=http://127.0.0.1:41605}
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 964253 T1876 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 964254 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 964254 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 964255 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 964255 T1878 C712 P41605 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 964272 T1833 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 964433 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 964434 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41605_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41605"}
[junit4:junit4] 2> 964447 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964465 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 964468 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:10317
[junit4:junit4] 2> 964468 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 964469 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 964469 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489
[junit4:junit4] 2> 964469 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/solr.xml
[junit4:junit4] 2> 964470 T1801 oasc.CoreContainer.<init> New CoreContainer 1052877441
[junit4:junit4] 2> 964470 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/'
[junit4:junit4] 2> 964470 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/'
[junit4:junit4] 2> 964574 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 964575 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 964575 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 964575 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 964576 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 964576 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 964576 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 964577 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 964577 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 964577 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 964589 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 964589 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 964589 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 964590 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964593 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a804510 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964594 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964596 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 964609 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964610 T1892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5db02d31 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964611 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964623 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 965273 T1833 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 965273 T1833 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:41605_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 965626 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:10317_
[junit4:junit4] 2> 965628 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:10317_
[junit4:junit4] 2> 965632 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 965632 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 965632 T1859 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965632 T1892 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965632 T1830 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965632 T1875 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965633 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965634 T1843 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965633 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965634 T1823 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965633 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965648 T1893 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 965648 T1893 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 965956 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 965957 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10317_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10317"}
[junit4:junit4] 2> 965957 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 965958 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 965973 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 966650 T1893 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1
[junit4:junit4] 2> 966650 T1893 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 966651 T1893 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 966651 T1893 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 966653 T1893 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/'
[junit4:junit4] 2> 966655 T1893 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/lib/README' to classloader
[junit4:junit4] 2> 966655 T1893 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 966710 T1893 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 966771 T1893 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 966872 T1893 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 966881 T1893 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C713 name=collection1 org.apache.solr.core.SolrCore@65177b81 url=http://127.0.0.1:41605/collection1 node=127.0.0.1:41605_ C713_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41605_, base_url=http://127.0.0.1:41605}
[junit4:junit4] 2> 967275 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:20986/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 967275 T1878 C713 P41605 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41605 START replicas=[http://127.0.0.1:20986/collection1/] nUpdates=100
[junit4:junit4] 2> 967276 T1878 C713 P41605 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 967277 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 967277 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 967277 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 967278 T1878 C713 P41605 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 967278 T1878 C713 P41605 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:20986/collection1/. core=collection1
[junit4:junit4] 2> 967278 T1878 C713 P41605 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C714 name=collection1 org.apache.solr.core.SolrCore@790af722 url=http://127.0.0.1:20986/collection1 node=127.0.0.1:20986_ C714_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:20986_, base_url=http://127.0.0.1:20986, leader=true}
[junit4:junit4] 2> 967286 T1834 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 967291 T1835 C714 P20986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 967294 T1835 C714 P20986 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 967295 T1835 C714 P20986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 967296 T1835 C714 P20986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_2,generation=2}
[junit4:junit4] 2> 967296 T1835 C714 P20986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 967297 T1835 C714 P20986 oass.SolrIndexSearcher.<init> Opening Searcher@7154583f realtime
[junit4:junit4] 2> 967297 T1835 C714 P20986 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 967298 T1835 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 967299 T1878 C713 P41605 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 967299 T1878 C713 P41605 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 967302 T1836 C714 P20986 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 967302 T1836 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 967303 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 967303 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 967303 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 967306 T1836 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 967307 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 967308 T1878 C713 P41605 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526
[junit4:junit4] 2> 967308 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d118d8c lockFactory=org.apache.lucene.store.NativeFSLockFactory@3405ec2e) fullCopy=false
[junit4:junit4] 2> 967311 T1836 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 967312 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 967313 T1878 C713 P41605 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 967313 T1878 C713 P41605 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 967316 T1878 C713 P41605 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 967316 T1878 C713 P41605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 967317 T1878 C713 P41605 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 967317 T1878 C713 P41605 oass.SolrIndexSearcher.<init> Opening Searcher@79c6e455 main
[junit4:junit4] 2> 967318 T1877 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c6e455 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 967319 T1878 C713 P41605 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526;done=true>>]
[junit4:junit4] 2> 967319 T1878 C713 P41605 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526
[junit4:junit4] 2> 967319 T1878 C713 P41605 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 967328 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 967445 T1893 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 967449 T1893 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 967452 T1893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 967457 T1893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 967480 T1893 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 967480 T1893 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/
[junit4:junit4] 2> 967481 T1893 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 967481 T1893 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 967482 T1893 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4
[junit4:junit4] 2> 967482 T1893 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index/
[junit4:junit4] 2> 967483 T1893 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 967483 T1893 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index
[junit4:junit4] 2> 967487 T1893 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_1,generation=1}
[junit4:junit4] 2> 967488 T1893 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 967491 T1893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 967491 T1893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 967492 T1893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 967493 T1893 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 967494 T1893 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 967494 T1893 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 967495 T1893 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 967495 T1893 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 967496 T1893 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 967497 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967498 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41605_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41605"}
[junit4:junit4] 2> 967512 T1893 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 967519 T1893 oass.SolrIndexSearcher.<init> Opening Searcher@c13d219 main
[junit4:junit4] 2> 967519 T1893 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/tlog
[junit4:junit4] 2> 967521 T1893 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 967521 T1893 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 967526 T1896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c13d219 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 967528 T1893 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 967528 T1893 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:10317 collection:collection1 shard:shard2
[junit4:junit4] 2> 967539 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967548 T1893 oasc.ZkController.register We are http://127.0.0.1:10317/collection1/ and leader is http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 967548 T1893 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:10317
[junit4:junit4] 2> 967549 T1893 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 967549 T1893 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C715 name=collection1 org.apache.solr.core.SolrCore@4fc79219 url=http://127.0.0.1:10317/collection1 node=127.0.0.1:10317_ C715_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:10317_, base_url=http://127.0.0.1:10317}
[junit4:junit4] 2> 967550 T1897 C715 P10317 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 967550 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 967550 T1893 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967550 T1897 C715 P10317 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 967551 T1897 C715 P10317 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 967551 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 967552 T1897 C715 P10317 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 967552 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 967553 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 967577 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967578 T1849 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 967578 T1801 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 967579 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 967579 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 968581 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 969050 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 969051 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10317_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10317"}
[junit4:junit4] 2> 969056 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969580 T1849 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 969580 T1849 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:10317_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 969583 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 970585 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C715_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:10317_, base_url=http://127.0.0.1:10317}
[junit4:junit4] 2> 971582 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:62105/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 971582 T1897 C715 P10317 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:10317 START replicas=[http://127.0.0.1:62105/collection1/] nUpdates=100
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 971584 T1897 C715 P10317 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 971584 T1897 C715 P10317 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:62105/collection1/. core=collection1
[junit4:junit4] 2> 971585 T1897 C715 P10317 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 971587 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C716 name=collection1 org.apache.solr.core.SolrCore@723241fd url=http://127.0.0.1:62105/collection1 node=127.0.0.1:62105_ C716_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:62105_, base_url=http://127.0.0.1:62105, leader=true}
[junit4:junit4] 2> 971592 T1850 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 971598 T1851 C716 P62105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 971600 T1851 C716 P62105 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_1,generation=1}
[junit4:junit4] 2> 971601 T1851 C716 P62105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 971602 T1851 C716 P62105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_2,generation=2}
[junit4:junit4] 2> 971603 T1851 C716 P62105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 971603 T1851 C716 P62105 oass.SolrIndexSearcher.<init> Opening Searcher@1b9fa58d realtime
[junit4:junit4] 2> 971604 T1851 C716 P62105 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 971604 T1851 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 971605 T1897 C715 P10317 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 971605 T1897 C715 P10317 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 971607 T1852 C716 P62105 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 971607 T1852 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 971608 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 971608 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 971608 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 971611 T1852 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 971612 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 971613 T1897 C715 P10317 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831
[junit4:junit4] 2> 971613 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@5b0dcf11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fc7f569) fullCopy=false
[junit4:junit4] 2> 971616 T1852 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 971617 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 971618 T1897 C715 P10317 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 971618 T1897 C715 P10317 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 971620 T1897 C715 P10317 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_2,generation=2}
[junit4:junit4] 2> 971620 T1897 C715 P10317 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 971620 T1897 C715 P10317 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 971621 T1897 C715 P10317 oass.SolrIndexSearcher.<init> Opening Searcher@2c294a8a main
[junit4:junit4] 2> 971622 T1896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c294a8a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831;done=true>>]
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 971624 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 971624 T1897 C715 P10317 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 971624 T1897 C715 P10317 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 971626 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 972075 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 972076 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10317_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10317"}
[junit4:junit4] 2> 972115 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972589 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 972590 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C717 name=collection1 org.apache.solr.core.SolrCore@29366d6b url=http://127.0.0.1:11430/collection1 node=127.0.0.1:11430_ C717_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:11430_, base_url=http://127.0.0.1:11430, leader=true}
[junit4:junit4] 2> 972599 T1813 C717 P11430 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972602 T1813 C717 P11430 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_1,generation=1}
[junit4:junit4] 2> 972603 T1813 C717 P11430 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 972603 T1813 C717 P11430 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_2,generation=2}
[junit4:junit4] 2> 972604 T1813 C717 P11430 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 972604 T1813 C717 P11430 oass.SolrIndexSearcher.<init> Opening Searcher@7d448d31 main
[junit4:junit4] 2> 972605 T1813 C717 P11430 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972606 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d448d31 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972607 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 972619 T1837 C714 P20986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972621 T1837 C714 P20986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972621 T1837 C714 P20986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972622 T1837 C714 P20986 oass.SolrIndexSearcher.<init> Opening Searcher@2f780015 main
[junit4:junit4] 2> 972623 T1837 C714 P20986 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972623 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f780015 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972624 T1837 C714 P20986 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:41605/collection1/, StdNode: http://127.0.0.1:62105/collection1/, StdNode: http://127.0.0.1:10317/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 972627 T1853 C716 P62105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972629 T1853 C716 P62105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972630 T1853 C716 P62105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972630 T1853 C716 P62105 oass.SolrIndexSearcher.<init> Opening Searcher@14c640c main
[junit4:junit4] 2> 972631 T1853 C716 P62105 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972632 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14c640c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972632 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> C713_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41605_, base_url=http://127.0.0.1:41605}
[junit4:junit4] 2> 972635 T1865 C713 P41605 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C715_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:10317_, base_url=http://127.0.0.1:10317}
[junit4:junit4] 2> 972635 T1882 C715 P10317 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972637 T1865 C713 P41605 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972637 T1882 C715 P10317 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972637 T1865 C713 P41605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972638 T1882 C715 P10317 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972639 T1865 C713 P41605 oass.SolrIndexSearcher.<init> Opening Searcher@774c3fd9 main
[junit4:junit4] 2> 972640 T1882 C715 P10317 oass.SolrIndexSearcher.<init> Opening Searcher@38cb4af0 main
[junit4:junit4] 2> 972640 T1865 C713 P41605 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972641 T1877 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@774c3fd9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972640 T1882 C715 P10317 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972641 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 972641 T1896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38cb4af0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972642 T1882 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 972643 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4] 2> 972644 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 972647 T1838 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 972649 T1866 C713 P41605 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 972651 T1854 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 972653 T1883 C715 P10317 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 974659 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437911283009585152)} 0 2
[junit4:junit4] 2> 974668 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1437911283013779456&update.from=http://127.0.0.1:62105/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437911283013779456)} 0 2
[junit4:junit4] 2> 974671 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1437911283017973760&update.from=http://127.0.0.1:20986/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437911283017973760)} 0 1
[junit4:junit4] 2> 974672 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1437911283017973760)} 0 6
[junit4:junit4] 2> 974673 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437911283013779456)} 0 12
[junit4:junit4] 2> 974677 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1437911283028459520)]} 0 1
[junit4:junit4] 2> 974684 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1437911283032653824)]} 0 1
[junit4:junit4] 2> 974684 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1437911283032653824)]} 0 4
[junit4:junit4] 2> 974687 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1437911283039993856)]} 0 0
[junit4:junit4] 2> 974695 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1437911283045236736)]} 0 0
[junit4:junit4] 2> 974696 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1437911283045236736)]} 0 4
[junit4:junit4] 2> 974697 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 7
[junit4:junit4] 2> 974699 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1437911283052576768)]} 0 0
[junit4:junit4] 2> 974706 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1437911283055722496)]} 0 1
[junit4:junit4] 2> 974706 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1437911283055722496)]} 0 4
[junit4:junit4] 2> 974709 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1437911283063062528)]} 0 0
[junit4:junit4] 2> 974715 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1437911283066208256)]} 0 0
[junit4:junit4] 2> 974716 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1437911283066208256)]} 0 4
[junit4:junit4] 2> 974719 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1437911283073548288)]} 0 0
[junit4:junit4] 2> 974727 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1437911283078791168)]} 0 0
[junit4:junit4] 2> 974728 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1437911283078791168)]} 0 4
[junit4:junit4] 2> 974729 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4]} 0 7
[junit4:junit4] 2> 974732 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1437911283086131200)]} 0 1
[junit4:junit4] 2> 974738 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1437911283089276928)]} 0 1
[junit4:junit4] 2> 974738 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1437911283089276928)]} 0 4
[junit4:junit4] 2> 974741 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1437911283096616960)]} 0 0
[junit4:junit4] 2> 974749 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1437911283101859840)]} 0 0
[junit4:junit4] 2> 974750 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1437911283101859840)]} 0 4
[junit4:junit4] 2> 974751 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6]} 0 7
[junit4:junit4] 2> 974754 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1437911283109199872)]} 0 1
[junit4:junit4] 2> 974761 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1437911283113394176)]} 0 1
[junit4:junit4] 2> 974761 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1437911283113394176)]} 0 4
[junit4:junit4] 2> 974765 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1437911283120734208)]} 0 1
[junit4:junit4] 2> 974771 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1437911283124928512)]} 0 1
[junit4:junit4] 2> 974771 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1437911283124928512)]} 0 4
[junit4:junit4] 2> 974774 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1437911283131219968)]} 0 0
[junit4:junit4] 2> 974781 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1437911283134365696)]} 0 1
[junit4:junit4] 2> 974781 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1437911283134365696)]} 0 4
[junit4:junit4] 2> 974785 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1437911283141705728)]} 0 1
[junit4:junit4] 2> 974791 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1437911283144851456)]} 0 1
[junit4:junit4] 2> 974791 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1437911283144851456)]} 0 4
[junit4:junit4] 2> 974794 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1437911283152191488)]} 0 0
[junit4:junit4] 2> 974800 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1437911283155337216)]} 0 0
[junit4:junit4] 2> 974801 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1437911283155337216)]} 0 4
[junit4:junit4] 2> 974804 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1437911283162677248)]} 0 0
[junit4:junit4] 2> 974812 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1437911283167920128)]} 0 0
[junit4:junit4] 2> 974813 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[12 (1437911283167920128)]} 0 4
[junit4:junit4] 2> 974814 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12]} 0 7
[junit4:junit4] 2> 974817 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1437911283176308736)]} 0 0
[junit4:junit4] 2> 974826 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1437911283181551616)]} 0 0
[junit4:junit4] 2> 974827 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1437911283181551616)]} 0 5
[junit4:junit4] 2> 974828 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13]} 0 8
[junit4:junit4] 2> 974831 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1437911283189940224)]} 0 1
[junit4:junit4] 2> 974838 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1437911283195183104)]} 0 0
[junit4:junit4] 2> 974839 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[14 (1437911283195183104)]} 0 4
[junit4:junit4] 2> 974840 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14]} 0 7
[junit4:junit4] 2> 974842 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1437911283202523136)]} 0 0
[junit4:junit4] 2> 974848 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1437911283205668864)]} 0 0
[junit4:junit4] 2> 974849 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1437911283205668864)]} 0 4
[junit4:junit4] 2> 974852 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1437911283213008896)]} 0 0
[junit4:junit4] 2> 974858 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1437911283216154624)]} 0 0
[junit4:junit4] 2> 974859 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1437911283216154624)]} 0 4
[junit4:junit4] 2> 974862 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1437911283223494656)]} 0 0
[junit4:junit4] 2> 974868 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1437911283226640384)]} 0 0
[junit4:junit4] 2> 974869 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1437911283226640384)]} 0 4
[junit4:junit4] 2> 974872 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1437911283233980416)]} 0 0
[junit4:junit4] 2> 974880 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1437911283239223296)]} 0 0
[junit4:junit4] 2> 974881 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1437911283239223296)]} 0 4
[junit4:junit4] 2> 974882 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18]} 0 7
[junit4:junit4] 2> 974884 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1437911283246563328)]} 0 0
[junit4:junit4] 2> 974890 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1437911283249709056)]} 0 0
[junit4:junit4] 2> 974891 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1437911283249709056)]} 0 4
[junit4:junit4] 2> 974895 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1437911283257049088)]} 0 1
[junit4:junit4] 2> 974900 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1437911283260194816)]} 0 0
[junit4:junit4] 2> 974901 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1437911283260194816)]} 0 4
[junit4:junit4] 2> 974904 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1437911283267534848)]} 0 0
[junit4:junit4] 2> 974912 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1437911283272777728)]} 0 0
[junit4:junit4] 2> 974913 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1437911283272777728)]} 0 4
[junit4:junit4] 2> 974914 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21]} 0 7
[junit4:junit4] 2> 974917 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1437911283280117760)]} 0 1
[junit4:junit4] 2> 974924 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1437911283285360640)]} 0 0
[junit4:junit4] 2> 974925 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[22 (1437911283285360640)]} 0 4
[junit4:junit4] 2> 974926 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22]} 0 7
[junit4:junit4] 2> 974928 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1437911283292700672)]} 0 0
[junit4:junit4] 2> 974934 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1437911283295846400)]} 0 0
[junit4:junit4] 2> 974935 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1437911283295846400)]} 0 4
[junit4:junit4] 2> 974938 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1437911283303186432)]} 0 0
[junit4:junit4] 2> 974944 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1437911283306332160)]} 0 0
[junit4:junit4] 2> 974945 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1437911283306332160)]} 0 4
[junit4:junit4] 2> 974948 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1437911283313672192)]} 0 0
[junit4:junit4] 2> 974956 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1437911283318915072)]} 0 0
[junit4:junit4] 2> 974957 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1437911283318915072)]} 0 4
[junit4:junit4] 2> 974958 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25]} 0 7
[junit4:junit4] 2> 974961 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1437911283326255104)]} 0 0
[junit4:junit4] 2> 974967 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1437911283329400832)]} 0 1
[junit4:junit4] 2> 974967 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1437911283329400832)]} 0 4
[junit4:junit4] 2> 974971 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1437911283336740864)]} 0 1
[junit4:junit4] 2> 974977 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1437911283339886592)]} 0 1
[junit4:junit4] 2> 974977 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1437911283339886592)]} 0 4
[junit4:junit4] 2> 974981 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1437911283347226624)]} 0 1
[junit4:junit4] 2> 974986 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1437911283350372352)]} 0 0
[junit4:junit4] 2> 974987 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1437911283350372352)]} 0 4
[junit4:junit4] 2> 974991 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1437911283357712384)]} 0 0
[junit4:junit4] 2> 974997 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1437911283360858112)]} 0 1
[junit4:junit4] 2> 974997 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1437911283360858112)]} 0 4
[junit4:junit4] 2> 975000 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1437911283368198144)]} 0 0
[junit4:junit4] 2> 975008 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1437911283373441024)]} 0 0
[junit4:junit4] 2> 975009 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1437911283373441024)]} 0 4
[junit4:junit4] 2> 975010 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30]} 0 7
[junit4:junit4] 2> 975012 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1437911283380781056)]} 0 0
[junit4:junit4] 2> 975018 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1437911283383926784)]} 0 0
[junit4:junit4] 2> 975019 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1437911283383926784)]} 0 4
[junit4:junit4] 2> 975022 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1437911283391266816)]} 0 0
[junit4:junit4] 2> 975030 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1437911283396509696)]} 0 0
[junit4:junit4] 2> 975031 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[32 (1437911283396509696)]} 0 4
[junit4:junit4] 2> 975032 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32]} 0 7
[junit4:junit4] 2> 975035 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1437911283403849728)]} 0 0
[junit4:junit4] 2> 975042 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1437911283409092608)]} 0 0
[junit4:junit4] 2> 975043 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1437911283409092608)]} 0 4
[junit4:junit4] 2> 975044 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33]} 0 7
[junit4:junit4] 2> 975047 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1437911283416432640)]} 0 1
[junit4:junit4] 2> 975054 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1437911283421675520)]} 0 0
[junit4:junit4] 2> 975055 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[34 (1437911283421675520)]} 0 4
[junit4:junit4] 2> 975056 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34]} 0 7
[junit4:junit4] 2> 975058 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1437911283429015552)]} 0 0
[junit4:junit4] 2> 975064 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1437911283432161280)]} 0 0
[junit4:junit4] 2> 975065 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1437911283432161280)]} 0 4
[junit4:junit4] 2> 975069 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1437911283439501312)]} 0 1
[junit4:junit4] 2> 975076 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1437911283444744192)]} 0 0
[junit4:junit4] 2> 975077 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[36 (1437911283444744192)]} 0 4
[junit4:junit4] 2> 975078 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36]} 0 7
[junit4:junit4] 2> 975080 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1437911283452084224)]} 0 0
[junit4:junit4] 2> 975088 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1437911283457327104)]} 0 0
[junit4:junit4] 2> 975089 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[37 (1437911283457327104)]} 0 4
[junit4:junit4] 2> 975090 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37]} 0 7
[junit4:junit4] 2> 975092 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1437911283464667136)]} 0 0
[junit4:junit4] 2> 975100 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1437911283469910016)]} 0 0
[junit4:junit4] 2> 975101 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[38 (1437911283469910016)]} 0 4
[junit4:junit4] 2> 975102 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38]} 0 7
[junit4:junit4] 2> 975104 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1437911283477250048)]} 0 0
[junit4:junit4] 2> 975110 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1437911283480395776)]} 0 0
[junit4:junit4] 2> 975111 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1437911283480395776)]} 0 4
[junit4:junit4] 2> 975114 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1437911283487735808)]} 0 0
[junit4:junit4] 2> 975122 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1437911283492978688)]} 0 0
[junit4:junit4] 2> 975123 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1437911283492978688)]} 0 4
[junit4:junit4] 2> 975124 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40]} 0 7
[junit4:junit4] 2> 975126 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1437911283500318720)]} 0 0
[junit4:junit4] 2> 975134 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1437911283505561600)]} 0 0
[junit4:junit4] 2> 975135 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[41 (1437911283505561600)]} 0 4
[junit4:junit4] 2> 975136 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41]} 0 7
[junit4:junit4] 2> 975139 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1437911283512901632)]} 0 1
[junit4:junit4] 2> 975146 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1437911283518144512)]} 0 0
[junit4:junit4] 2> 975147 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[42 (1437911283518144512)]} 0 4
[junit4:junit4] 2> 975148 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42]} 0 7
[junit4:junit4] 2> 975150 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1437911283525484544)]} 0 0
[junit4:junit4] 2> 975156 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1437911283528630272)]} 0 0
[junit4:junit4] 2> 975157 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1437911283528630272)]} 0 4
[junit4:junit4] 2> 975160 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1437911283535970304)]} 0 0
[junit4:junit4] 2> 975168 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1437911283541213184)]} 0 0
[junit4:junit4] 2> 975169 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[44 (1437911283541213184)]} 0 4
[junit4:junit4] 2> 975170 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44]} 0 7
[junit4:junit4] 2> 975172 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1437911283548553216)]} 0 0
[junit4:junit4] 2> 975178 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1437911283551698944)]} 0 0
[junit4:junit4] 2> 975179 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1437911283551698944)]} 0 4
[junit4:junit4] 2> 975182 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1437911283559038976)]} 0 0
[junit4:junit4] 2> 975189 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1437911283562184704)]} 0 1
[junit4:junit4] 2> 975189 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1437911283562184704)]} 0 4
[junit4:junit4] 2> 975192 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1437911283569524736)]} 0 0
[junit4:junit4] 2> 975200 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1437911283574767616)]} 0 0
[junit4:junit4] 2> 975201 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1437911283574767616)]} 0 4
[junit4:junit4] 2> 975202 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47]} 0 7
[junit4:junit4] 2> 975205 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1437911283582107648)]} 0 1
[junit4:junit4] 2> 975210 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1437911283585253376)]} 0 0
[junit4:junit4] 2> 975211 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1437911283585253376)]} 0 4
[junit4:junit4] 2> 975215 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1437911283592593408)]} 0 1
[junit4:junit4] 2> 975220 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1437911283596787712)]} 0 0
[junit4:junit4] 2> 975221 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1437911283596787712)]} 0 4
[junit4:junit4] 2> 975225 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1437911283603079168)]} 0 1
[junit4:junit4] 2> 975232 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1437911283608322048)]} 0 0
[junit4:junit4] 2> 975233 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[50 (1437911283608322048)]} 0 4
[junit4:junit4] 2> 975234 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50]} 0 7
[junit4:junit4] 2> 975236 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1437911283615662080)]} 0 0
[junit4:junit4] 2> 975244 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1437911283620904960)]} 0 0
[junit4:junit4] 2> 975245 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[51 (1437911283620904960)]} 0 4
[junit4:junit4] 2> 975246 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51]} 0 7
[junit4:junit4] 2> 975248 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1437911283628244992)]} 0 0
[junit4:junit4] 2> 975256 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1437911283633487872)]} 0 0
[junit4:junit4] 2> 975257 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[52 (1437911283633487872)]} 0 4
[junit4:junit4] 2> 975258 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52]} 0 7
[junit4:junit4] 2> 975260 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1437911283640827904)]} 0 0
[junit4:junit4] 2> 975268 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1437911283646070784)]} 0 0
[junit4:junit4] 2> 975269 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[53 (1437911283646070784)]} 0 4
[junit4:junit4] 2> 975270 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53]} 0 7
[junit4:junit4] 2> 975272 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1437911283653410816)]} 0 0
[junit4:junit4] 2> 975278 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1437911283656556544)]} 0 0
[junit4:junit4] 2> 975279 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1437911283656556544)]} 0 4
[junit4:junit4] 2> 975282 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1437911283663896576)]} 0 0
[junit4:junit4] 2> 975290 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1437911283669139456)]} 0 0
[junit4:junit4] 2> 975291 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1437911283669139456)]} 0 4
[junit4:junit4] 2> 975292 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55]} 0 7
[junit4:junit4] 2> 975295 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1437911283676479488)]} 0 1
[junit4:junit4] 2> 975300 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1437911283679625216)]} 0 0
[junit4:junit4] 2> 975301 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1437911283679625216)]} 0 4
[junit4:junit4] 2> 975304 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1437911283686965248)]} 0 0
[junit4:junit4] 2> 975312 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1437911283692208128)]} 0 0
[junit4:junit4] 2> 975313 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[57 (1437911283692208128)]} 0 4
[junit4:junit4] 2> 975314 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57]} 0 7
[junit4:junit4] 2> 975316 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1437911283699548160)]} 0 0
[junit4:junit4] 2> 975324 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1437911283704791040)]} 0 0
[junit4:junit4] 2> 975325 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[58 (1437911283704791040)]} 0 4
[junit4:junit4] 2> 975326 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58]} 0 7
[junit4:junit4] 2> 975328 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1437911283712131072)]} 0 0
[junit4:junit4] 2> 975336 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1437911283717373952)]} 0 0
[junit4:junit4] 2> 975337 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[59 (1437911283717373952)]} 0 4
[junit4:junit4] 2> 975338 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59]} 0 7
[junit4:junit4] 2> 975340 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1437911283724713984)]} 0 0
[junit4:junit4] 2> 975346 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1437911283727859712)]} 0 0
[junit4:junit4] 2> 975347 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1437911283727859712)]} 0 4
[junit4:junit4] 2> 975352 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1437911283736248320)]} 0 1
[junit4:junit4] 2> 975359 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1437911283741491200)]} 0 0
[junit4:junit4] 2> 975360 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[61 (1437911283741491200)]} 0 4
[junit4:junit4] 2> 975361 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61]} 0 7
[junit4:junit4] 2> 975363 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1437911283748831232)]} 0 0
[junit4:junit4] 2> 975369 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1437911283751976960)]} 0 0
[junit4:junit4] 2> 975370 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1437911283751976960)]} 0 4
[junit4:junit4] 2> 975374 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1437911283759316992)]} 0 0
[junit4:junit4] 2> 975379 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1437911283762462720)]} 0 0
[junit4:junit4] 2> 975380 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1437911283762462720)]} 0 4
[junit4:junit4] 2> 975383 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1437911283769802752)]} 0 0
[junit4:junit4] 2> 975392 T1865 C713 P
[...truncated too long message...]
updating... (live nodes size: 6)
[junit4:junit4] 2> 1402410 T2155 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1402410 T2155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 1402410 T2155 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1402410 T2155 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1405637 T2212 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:23965 couldn't connect to http://127.0.0.1:39884/collection1/, counting as success
[junit4:junit4] 2> 1405637 T2212 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:23965 DONE. sync succeeded
[junit4:junit4] 2> 1405638 T2212 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 1405638 T2212 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4:junit4] 2> 1405639 T2212 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1e4a067e
[junit4:junit4] 2> 1405640 T2212 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=941,cumulative_deletesById=476,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 1405641 T2212 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1405641 T2212 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1405641 T2212 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1405644 T2212 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1405645 T2212 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1405645 T2212 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12;done=false>>]
[junit4:junit4] 2> 1405646 T2212 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12
[junit4:junit4] 2> 1405646 T2212 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12/index;done=false>>]
[junit4:junit4] 2> 1405646 T2212 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12/index
[junit4:junit4] 2> 1405646 T2212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1405647 T2212 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1405647 T2212 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1405647 T2212 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=in, timezone=US/Eastern
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=241443848,total=529530880
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestNumberUtils, NoCacheHeaderTest, TestQuerySenderNoQuery, TestSuggestSpellingConverter, HighlighterConfigTest, TestDFRSimilarityFactory, LeaderElectionIntegrationTest, HardAutoCommitTest, RequestHandlersTest, SolrPluginUtilsTest, TestUtils, TestSolrQueryParserResource, UpdateParamsTest, NotRequiredUniqueKeyTest, IndexSchemaRuntimeFieldTest, DirectUpdateHandlerTest, OpenExchangeRatesOrgProviderTest, TestComponentsName, StandardRequestHandlerTest, TestRandomFaceting, TestElisionMultitermQuery, TestDistributedGrouping, ShardRoutingCustomTest, TestIndexingPerformance, BasicDistributedZk2Test, SimplePostToolTest, TestSolrQueryParserDefaultOperatorResource, TermVectorComponentDistributedTest, PreAnalyzedUpdateProcessorTest, CopyFieldTest, RequiredFieldsTest, TestPropInject, CollectionsAPIDistributedZkTest, TestDynamicFieldCollectionResource, SolrCoreCheckLockOnStartupTest, TestPostingsSolrHighlighter, TestHashPartitioner, TestFieldResource, TestFastOutputStream, TestMergePolicyConfig, DocumentAnalysisRequestHandlerTest, DebugComponentTest, SuggesterTSTTest, TestLazyCores, TestDistributedSearch, XsltUpdateRequestHandlerTest, TestAtomicUpdateErrorCases, TestCodecSupport, DOMUtilTest, TestQueryTypes, BasicZkTest, ChaosMonkeyShardSplitTest, TestJmxIntegration, XmlUpdateRequestHandlerTest, PingRequestHandlerTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 261.77s, 1 test, 1 failure <<< FAILURES!
[...truncated 518 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:386: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:366: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:887: There were test failures: 296 suites, 1243 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 54 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure