You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/06/14 08:19:22 UTC
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4064 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4064/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 44 from http://127.0.0.1:40990/nikd/collection1lastClient and got 42 from http://127.0.0.1:44131/nikd/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 44 from http://127.0.0.1:40990/nikd/collection1lastClient and got 42 from http://127.0.0.1:44131/nikd/collection1
at __randomizedtesting.SeedInfo.seed([8CF332BB011E671E:D15BCA376410722]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9473 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 792527 T828 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nikd/
[junit4:junit4] 2> 792535 T828 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371189629131
[junit4:junit4] 2> 792537 T828 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 792537 T829 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 792638 T828 oasc.ZkTestServer.run start zk server on port:46037
[junit4:junit4] 2> 792639 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 792645 T835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e624b9c name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 792645 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 792645 T828 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 792668 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 792674 T837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c78f435 name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 792675 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 792675 T828 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 792679 T828 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 792687 T828 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 792689 T828 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 792697 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 792698 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 792711 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 792713 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 792820 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 792821 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 792825 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 792825 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 792837 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 792837 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 792841 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 792841 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 792845 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 792846 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 792849 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 792850 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 792853 T828 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 792854 T828 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 793046 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 793050 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:24743
[junit4:junit4] 2> 793051 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 793051 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 793051 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459
[junit4:junit4] 2> 793052 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/solr.xml
[junit4:junit4] 2> 793052 T828 oasc.CoreContainer.<init> New CoreContainer 1227673543
[junit4:junit4] 2> 793052 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/'
[junit4:junit4] 2> 793053 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/'
[junit4:junit4] 2> 793138 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 793139 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 793139 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 793140 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 793140 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 793140 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 793141 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 793141 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 793141 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 793142 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 793154 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 793154 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 793155 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 793155 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 793158 T848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26eccef name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 793159 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 793172 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 793186 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 793188 T850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c0982df name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 793188 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 793191 T828 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 793201 T828 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 793204 T828 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 793207 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24743_nikd
[junit4:junit4] 2> 793213 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24743_nikd
[junit4:junit4] 2> 793216 T828 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 793237 T828 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 793240 T828 oasc.Overseer.start Overseer (id=89862283535253507-127.0.0.1:24743_nikd-n_0000000000) starting
[junit4:junit4] 2> 793245 T828 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 793260 T852 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 793261 T828 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 793263 T828 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 793271 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 793273 T851 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 793278 T853 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 793278 T853 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 794776 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 794777 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:24743_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:24743/nikd"}
[junit4:junit4] 2> 794777 T851 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 794778 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 794793 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 795281 T853 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/collection1
[junit4:junit4] 2> 795281 T853 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 795282 T853 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 795282 T853 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 795283 T853 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/collection1/'
[junit4:junit4] 2> 795284 T853 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/collection1/lib/README' to classloader
[junit4:junit4] 2> 795285 T853 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 795324 T853 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 795370 T853 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 795471 T853 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 795479 T853 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 796171 T853 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 796187 T853 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 796191 T853 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 796215 T853 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 796221 T853 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 796225 T853 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 796227 T853 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 796227 T853 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 796228 T853 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 796229 T853 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 796229 T853 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 796229 T853 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 796280 T853 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371189629459/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/control/data/
[junit4:junit4] 2> 796280 T853 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 796281 T853 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 796282 T853 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/control/data
[junit4:junit4] 2> 796283 T853 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/control/data/index/
[junit4:junit4] 2> 796284 T853 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 796285 T853 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/control/data/index
[junit4:junit4] 2> 796292 T853 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 796292 T853 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 796295 T853 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 796296 T853 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 796296 T853 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 796297 T853 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 796297 T853 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 796298 T853 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 796298 T853 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 796299 T853 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 796299 T853 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 796315 T853 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 796325 T853 oass.SolrIndexSearcher.<init> Opening Searcher@7d8d0173 main
[junit4:junit4] 2> 796326 T853 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 796327 T853 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 796331 T854 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d8d0173 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 796333 T853 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 796333 T853 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:24743/nikd collection:control_collection shard:shard1
[junit4:junit4] 2> 796334 T853 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 796436 T853 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 796469 T853 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 796469 T853 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 796469 T853 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:24743/nikd/collection1/
[junit4:junit4] 2> 796470 T853 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 796470 T853 oasc.SyncStrategy.syncToMe http://127.0.0.1:24743/nikd/collection1/ has no replicas
[junit4:junit4] 2> 796470 T853 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:24743/nikd/collection1/
[junit4:junit4] 2> 796470 T853 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 797800 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 797825 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 797857 T853 oasc.ZkController.register We are http://127.0.0.1:24743/nikd/collection1/ and leader is http://127.0.0.1:24743/nikd/collection1/
[junit4:junit4] 2> 797857 T853 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:24743/nikd
[junit4:junit4] 2> 797857 T853 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 797858 T853 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 797858 T853 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 797860 T853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 797862 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 797862 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 797863 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 797882 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 797884 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 797886 T857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6393d737 name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 797887 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 797889 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 797891 T828 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 798146 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 798149 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32712
[junit4:junit4] 2> 798150 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 798151 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 798151 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486
[junit4:junit4] 2> 798151 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/solr.xml
[junit4:junit4] 2> 798152 T828 oasc.CoreContainer.<init> New CoreContainer 1221237268
[junit4:junit4] 2> 798153 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/'
[junit4:junit4] 2> 798153 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/'
[junit4:junit4] 2> 798273 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 798274 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 798274 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 798275 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 798275 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 798276 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 798276 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 798277 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 798277 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 798278 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 798295 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 798296 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 798296 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 798297 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 798300 T868 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f382eae name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 798301 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 798310 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 798323 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 798325 T870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10dc0cb4 name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 798325 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 798337 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 799340 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32712_nikd
[junit4:junit4] 2> 799341 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 799352 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32712_nikd
[junit4:junit4] 2> 799352 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:24743_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:24743/nikd"}
[junit4:junit4] 2> 799367 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 799368 T857 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 799368 T870 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 799370 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 799370 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 799383 T850 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 799384 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 799387 T871 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 799387 T871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 800884 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 800885 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32712_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32712/nikd"}
[junit4:junit4] 2> 800886 T851 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 800886 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 800937 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 800937 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 800937 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 801391 T871 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/collection1
[junit4:junit4] 2> 801391 T871 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 801392 T871 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 801392 T871 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 801394 T871 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/collection1/'
[junit4:junit4] 2> 801396 T871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 801396 T871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/collection1/lib/README' to classloader
[junit4:junit4] 2> 801452 T871 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 801516 T871 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 801618 T871 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 801627 T871 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 802303 T871 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 802317 T871 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 802321 T871 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 802338 T871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 802343 T871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 802348 T871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 802349 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 802349 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 802350 T871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 802351 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 802351 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 802351 T871 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 802352 T871 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371189634486/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/
[junit4:junit4] 2> 802352 T871 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 802353 T871 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 802354 T871 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1
[junit4:junit4] 2> 802355 T871 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index/
[junit4:junit4] 2> 802356 T871 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 802357 T871 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index
[junit4:junit4] 2> 802365 T871 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 802365 T871 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 802369 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 802369 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 802370 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 802371 T871 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 802371 T871 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 802372 T871 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 802372 T871 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 802373 T871 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 802373 T871 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 802392 T871 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 802403 T871 oass.SolrIndexSearcher.<init> Opening Searcher@736642a1 main
[junit4:junit4] 2> 802404 T871 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 802404 T871 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 802410 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@736642a1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 802412 T871 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 802413 T871 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32712/nikd collection:collection1 shard:shard1
[junit4:junit4] 2> 802414 T871 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 802431 T871 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 802443 T871 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 802443 T871 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 802443 T871 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32712/nikd/collection1/
[junit4:junit4] 2> 802444 T871 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 802444 T871 oasc.SyncStrategy.syncToMe http://127.0.0.1:32712/nikd/collection1/ has no replicas
[junit4:junit4] 2> 802444 T871 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32712/nikd/collection1/
[junit4:junit4] 2> 802445 T871 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 803945 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 803964 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 803964 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 803964 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 803990 T871 oasc.ZkController.register We are http://127.0.0.1:32712/nikd/collection1/ and leader is http://127.0.0.1:32712/nikd/collection1/
[junit4:junit4] 2> 803990 T871 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32712/nikd
[junit4:junit4] 2> 803990 T871 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 803990 T871 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 803991 T871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 803993 T871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 803994 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 803994 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 803995 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 804293 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 804297 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:10744
[junit4:junit4] 2> 804297 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 804298 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 804299 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609
[junit4:junit4] 2> 804299 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/solr.xml
[junit4:junit4] 2> 804300 T828 oasc.CoreContainer.<init> New CoreContainer 559108273
[junit4:junit4] 2> 804300 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/'
[junit4:junit4] 2> 804301 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/'
[junit4:junit4] 2> 804446 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 804446 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 804447 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 804447 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 804448 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 804448 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 804449 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 804449 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 804450 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 804450 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 804468 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 804469 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 804469 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 804470 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 804474 T884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a0a874a name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 804474 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 804491 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 804503 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 804505 T886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff04a16 name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 804505 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 804516 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 805470 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 805471 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32712_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32712/nikd"}
[junit4:junit4] 2> 805487 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 805487 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 805488 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 805487 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 805519 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:10744_nikd
[junit4:junit4] 2> 805521 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:10744_nikd
[junit4:junit4] 2> 805523 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 805524 T857 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 805524 T870 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 805524 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 805524 T886 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 805524 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 805525 T850 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 805525 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 805537 T887 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 805537 T887 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 806993 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 806995 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10744_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10744/nikd"}
[junit4:junit4] 2> 806995 T851 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 806995 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 807042 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 807042 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 807042 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 807042 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 807541 T887 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/collection1
[junit4:junit4] 2> 807541 T887 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 807542 T887 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 807542 T887 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 807544 T887 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/collection1/'
[junit4:junit4] 2> 807546 T887 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/collection1/lib/README' to classloader
[junit4:junit4] 2> 807546 T887 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 807600 T887 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 807662 T887 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 807763 T887 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 807771 T887 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 808420 T887 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 808430 T887 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 808433 T887 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 808446 T887 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 808450 T887 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 808454 T887 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 808455 T887 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 808455 T887 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 808455 T887 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 808456 T887 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 808457 T887 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 808457 T887 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 808457 T887 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371189640609/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/
[junit4:junit4] 2> 808457 T887 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 808458 T887 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 808459 T887 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2
[junit4:junit4] 2> 808459 T887 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index/
[junit4:junit4] 2> 808460 T887 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 808461 T887 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index
[junit4:junit4] 2> 808466 T887 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 808466 T887 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 808470 T887 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 808470 T887 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 808471 T887 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 808471 T887 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 808472 T887 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 808472 T887 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 808472 T887 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 808473 T887 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 808473 T887 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 808484 T887 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 808491 T887 oass.SolrIndexSearcher.<init> Opening Searcher@714fa4b7 main
[junit4:junit4] 2> 808492 T887 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 808492 T887 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 808497 T888 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@714fa4b7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 808499 T887 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 808499 T887 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:10744/nikd collection:collection1 shard:shard2
[junit4:junit4] 2> 808500 T887 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 808515 T887 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 808526 T887 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 808526 T887 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 808526 T887 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:10744/nikd/collection1/
[junit4:junit4] 2> 808527 T887 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 808527 T887 oasc.SyncStrategy.syncToMe http://127.0.0.1:10744/nikd/collection1/ has no replicas
[junit4:junit4] 2> 808527 T887 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:10744/nikd/collection1/
[junit4:junit4] 2> 808528 T887 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 808547 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 808561 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 808561 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 808561 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 808561 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 808590 T887 oasc.ZkController.register We are http://127.0.0.1:10744/nikd/collection1/ and leader is http://127.0.0.1:10744/nikd/collection1/
[junit4:junit4] 2> 808590 T887 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:10744/nikd
[junit4:junit4] 2> 808590 T887 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 808590 T887 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 808591 T887 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 808592 T887 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 808593 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 808594 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 808595 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 808796 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 808798 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32169
[junit4:junit4] 2> 808799 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 808799 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 808799 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202
[junit4:junit4] 2> 808800 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/solr.xml
[junit4:junit4] 2> 808800 T828 oasc.CoreContainer.<init> New CoreContainer 646495945
[junit4:junit4] 2> 808801 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/'
[junit4:junit4] 2> 808801 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/'
[junit4:junit4] 2> 808896 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 808896 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 808897 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 808897 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 808897 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 808898 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 808898 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 808898 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 808899 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 808899 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 808911 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 808912 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 808912 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 808913 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 808916 T900 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75f561c9 name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 808917 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 808931 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 808945 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 808947 T902 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1680dcde name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 808948 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 808956 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 809959 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32169_nikd
[junit4:junit4] 2> 809961 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32169_nikd
[junit4:junit4] 2> 809964 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 809965 T870 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 809965 T857 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 809965 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 809965 T886 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 809965 T902 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 809966 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 809966 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 809966 T850 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 809981 T903 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 809981 T903 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 810066 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 810067 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10744_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10744/nikd"}
[junit4:junit4] 2> 810071 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32169_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32169/nikd"}
[junit4:junit4] 2> 810072 T851 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 810072 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 810088 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 810088 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 810088 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 810088 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 810088 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 810983 T903 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/collection1
[junit4:junit4] 2> 810983 T903 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 810984 T903 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 810984 T903 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 810986 T903 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/collection1/'
[junit4:junit4] 2> 810987 T903 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 810988 T903 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/collection1/lib/README' to classloader
[junit4:junit4] 2> 811041 T903 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 811104 T903 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 811206 T903 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 811215 T903 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 811889 T903 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 811901 T903 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 811904 T903 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 811917 T903 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 811920 T903 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 811924 T903 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 811925 T903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 811926 T903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 811926 T903 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 811927 T903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 811927 T903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 811927 T903 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 811928 T903 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371189645202/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty3/
[junit4:junit4] 2> 811928 T903 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 811928 T903 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 811929 T903 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty3
[junit4:junit4] 2> 811930 T903 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty3/index/
[junit4:junit4] 2> 811930 T903 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 811931 T903 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty3/index
[junit4:junit4] 2> 811944 T903 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 811945 T903 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 811947 T903 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 811947 T903 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 811948 T903 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 811948 T903 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 811949 T903 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 811949 T903 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 811949 T903 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 811950 T903 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 811950 T903 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 811963 T903 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 811970 T903 oass.SolrIndexSearcher.<init> Opening Searcher@45e1fd8 main
[junit4:junit4] 2> 811971 T903 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 811971 T903 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 811975 T904 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45e1fd8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 811976 T903 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 811977 T903 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32169/nikd collection:collection1 shard:shard3
[junit4:junit4] 2> 811978 T903 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 811997 T903 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 812008 T903 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 812008 T903 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 812008 T903 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32169/nikd/collection1/
[junit4:junit4] 2> 812008 T903 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 812009 T903 oasc.SyncStrategy.syncToMe http://127.0.0.1:32169/nikd/collection1/ has no replicas
[junit4:junit4] 2> 812009 T903 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32169/nikd/collection1/
[junit4:junit4] 2> 812009 T903 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 813096 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 813117 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 813117 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 813117 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 813117 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 813117 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 813144 T903 oasc.ZkController.register We are http://127.0.0.1:32169/nikd/collection1/ and leader is http://127.0.0.1:32169/nikd/collection1/
[junit4:junit4] 2> 813144 T903 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32169/nikd
[junit4:junit4] 2> 813144 T903 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 813145 T903 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 813145 T903 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 813147 T903 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 813148 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 813149 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 813149 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 813451 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 813454 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:17667
[junit4:junit4] 2> 813455 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 813455 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 813456 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764
[junit4:junit4] 2> 813456 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/solr.xml
[junit4:junit4] 2> 813457 T828 oasc.CoreContainer.<init> New CoreContainer 184679244
[junit4:junit4] 2> 813458 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/'
[junit4:junit4] 2> 813458 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/'
[junit4:junit4] 2> 813590 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 813590 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 813591 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 813591 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 813592 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 813592 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 813593 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 813593 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 813594 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 813594 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 813612 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 813613 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 813613 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 813614 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 813618 T916 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37da2070 name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 813619 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 813634 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 813644 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 813647 T918 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50ce07de name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 813647 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 813656 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 814628 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 814629 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32169_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32169/nikd"}
[junit4:junit4] 2> 814642 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814642 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814642 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814642 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814642 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814643 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814660 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17667_nikd
[junit4:junit4] 2> 814662 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17667_nikd
[junit4:junit4] 2> 814664 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814664 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 814665 T857 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 814665 T902 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 814665 T870 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 814665 T886 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 814666 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 814665 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 814665 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 814666 T918 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 814666 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 814671 T850 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 814678 T919 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 814678 T919 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 816148 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 816149 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:17667_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:17667/nikd"}
[junit4:junit4] 2> 816149 T851 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 816149 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 816164 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 816164 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 816164 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 816164 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 816164 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 816164 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 816682 T919 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/collection1
[junit4:junit4] 2> 816682 T919 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 816684 T919 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 816684 T919 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 816686 T919 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/collection1/'
[junit4:junit4] 2> 816688 T919 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/collection1/lib/README' to classloader
[junit4:junit4] 2> 816688 T919 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 816745 T919 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 816825 T919 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 816827 T919 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 816835 T919 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 817457 T919 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 817467 T919 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 817470 T919 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 817483 T919 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 817487 T919 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 817490 T919 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 817491 T919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 817492 T919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 817492 T919 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 817493 T919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 817493 T919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 817493 T919 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 817493 T919 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371189649764/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/
[junit4:junit4] 2> 817494 T919 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 817494 T919 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 817495 T919 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4
[junit4:junit4] 2> 817495 T919 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index/
[junit4:junit4] 2> 817496 T919 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 817497 T919 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index
[junit4:junit4] 2> 817501 T919 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 817501 T919 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 817504 T919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 817504 T919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 817505 T919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 817505 T919 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 817506 T919 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 817506 T919 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 817506 T919 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 817507 T919 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 817507 T919 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 817518 T919 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 817525 T919 oass.SolrIndexSearcher.<init> Opening Searcher@19aa78e0 main
[junit4:junit4] 2> 817526 T919 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 817526 T919 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 817530 T920 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19aa78e0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 817532 T919 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 817533 T919 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:17667/nikd collection:collection1 shard:shard1
[junit4:junit4] 2> 817536 T919 oasc.ZkController.register We are http://127.0.0.1:17667/nikd/collection1/ and leader is http://127.0.0.1:32712/nikd/collection1/
[junit4:junit4] 2> 817536 T919 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:17667/nikd
[junit4:junit4] 2> 817536 T919 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 817537 T919 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C321 name=collection1 org.apache.solr.core.SolrCore@2f5c3732 url=http://127.0.0.1:17667/nikd/collection1 node=127.0.0.1:17667_nikd C321_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:17667_nikd, base_url=http://127.0.0.1:17667/nikd}
[junit4:junit4] 2> 817537 T921 C321 P17667 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 817537 T919 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 817537 T921 C321 P17667 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 817538 T921 C321 P17667 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 817538 T921 C321 P17667 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 817538 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 817539 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 817539 T921 C321 P17667 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 817540 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 817559 T860 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 817668 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 817669 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:17667_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:17667/nikd"}
[junit4:junit4] 2> 817681 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 817681 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 817681 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 817681 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 817681 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 817681 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 817817 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 817820 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:49961
[junit4:junit4] 2> 817821 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 817821 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 817822 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153
[junit4:junit4] 2> 817822 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/solr.xml
[junit4:junit4] 2> 817823 T828 oasc.CoreContainer.<init> New CoreContainer 272899076
[junit4:junit4] 2> 817824 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/'
[junit4:junit4] 2> 817824 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/'
[junit4:junit4] 2> 817945 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 817946 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 817947 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 817947 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 817948 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 817948 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 817949 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 817949 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 817949 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 817950 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 817968 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 817969 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 817969 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 817970 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 817973 T933 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@580d288f name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 817973 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 817976 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 817989 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 817994 T935 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6597de4e name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 817994 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 818006 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 818560 T860 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 818560 T860 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:17667_nikd&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 819009 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49961_nikd
[junit4:junit4] 2> 819012 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49961_nikd
[junit4:junit4] 2> 819015 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 819015 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 819016 T857 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819016 T935 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819016 T870 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819016 T902 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819016 T886 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819017 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819017 T918 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819017 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819016 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819017 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819017 T850 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 819032 T936 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 819032 T936 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 819190 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 819191 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49961_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49961/nikd"}
[junit4:junit4] 2> 819192 T851 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 819192 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 819206 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819206 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819206 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819206 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819206 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819206 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 819206 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820035 T936 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/collection1
[junit4:junit4] 2> 820035 T936 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 820036 T936 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 820036 T936 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 820038 T936 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/collection1/'
[junit4:junit4] 2> 820040 T936 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 820040 T936 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/collection1/lib/README' to classloader
[junit4:junit4] 2> 820096 T936 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 820172 T936 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 820273 T936 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 820280 T936 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C322 name=collection1 org.apache.solr.core.SolrCore@2f5c3732 url=http://127.0.0.1:17667/nikd/collection1 node=127.0.0.1:17667_nikd C322_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:17667_nikd, base_url=http://127.0.0.1:17667/nikd}
[junit4:junit4] 2> 820562 T921 C322 P17667 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:32712/nikd/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 820562 T921 C322 P17667 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:17667/nikd START replicas=[http://127.0.0.1:32712/nikd/collection1/] nUpdates=100
[junit4:junit4] 2> 820563 T921 C322 P17667 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 820564 T921 C322 P17667 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 820564 T921 C322 P17667 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 820565 T921 C322 P17667 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 820565 T921 C322 P17667 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 820565 T921 C322 P17667 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:32712/nikd/collection1/. core=collection1
[junit4:junit4] 2> 820565 T921 C322 P17667 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C323 name=collection1 org.apache.solr.core.SolrCore@1333e6e6 url=http://127.0.0.1:32712/nikd/collection1 node=127.0.0.1:32712_nikd C323_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:32712_nikd, base_url=http://127.0.0.1:32712/nikd, leader=true}
[junit4:junit4] 2> 820574 T861 C323 P32712 oasc.SolrCore.execute [collection1] webapp=/nikd path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 820579 T862 C323 P32712 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 820586 T862 C323 P32712 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 820586 T862 C323 P32712 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 820590 T862 C323 P32712 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 820591 T862 C323 P32712 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 820593 T862 C323 P32712 oass.SolrIndexSearcher.<init> Opening Searcher@446580d4 realtime
[junit4:junit4] 2> 820593 T862 C323 P32712 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 820594 T862 C323 P32712 oasup.LogUpdateProcessor.finish [collection1] webapp=/nikd path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 820595 T921 C322 P17667 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 820595 T921 C322 P17667 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 820598 T863 C323 P32712 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 820598 T863 C323 P32712 oasc.SolrCore.execute [collection1] webapp=/nikd path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 820599 T921 C322 P17667 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 820600 T921 C322 P17667 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 820600 T921 C322 P17667 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 820603 T863 C323 P32712 oasc.SolrCore.execute [collection1] webapp=/nikd path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 820604 T921 C322 P17667 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 820605 T921 C322 P17667 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index.20130614010057199
[junit4:junit4] 2> 820606 T921 C322 P17667 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index.20130614010057199 lockFactory=org.apache.lucene.store.NativeFSLockFactory@10d54514 fullCopy=false
[junit4:junit4] 2> 820610 T863 C323 P32712 oasc.SolrCore.execute [collection1] webapp=/nikd path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 820612 T921 C322 P17667 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 820615 T921 C322 P17667 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 820616 T921 C322 P17667 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 820622 T921 C322 P17667 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 820622 T921 C322 P17667 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 820623 T921 C322 P17667 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 820624 T921 C322 P17667 oass.SolrIndexSearcher.<init> Opening Searcher@52721b3a main
[junit4:junit4] 2> 820626 T920 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@52721b3a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 820627 T921 C322 P17667 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index.20130614010057199 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index.20130614010057199;done=true>>]
[junit4:junit4] 2> 820627 T921 C322 P17667 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index.20130614010057199
[junit4:junit4] 2> 820627 T921 C322 P17667 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty4/index.20130614010057199
[junit4:junit4] 2> 820629 T921 C322 P17667 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 820629 T921 C322 P17667 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 820630 T921 C322 P17667 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 820630 T921 C322 P17667 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 820632 T921 C322 P17667 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 820711 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 820712 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:17667_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:17667/nikd"}
[junit4:junit4] 2> 820726 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820726 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820726 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820726 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820726 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820726 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820726 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 820940 T936 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 820950 T936 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 820953 T936 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 820966 T936 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 820980 T936 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 820983 T936 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 820984 T936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 820985 T936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 820985 T936 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 820986 T936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 820986 T936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 820987 T936 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 820987 T936 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371189654153/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/
[junit4:junit4] 2> 820987 T936 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 820988 T936 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 820988 T936 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5
[junit4:junit4] 2> 820989 T936 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index/
[junit4:junit4] 2> 820990 T936 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 820990 T936 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index
[junit4:junit4] 2> 820995 T936 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 820995 T936 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 820998 T936 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 820998 T936 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 820999 T936 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 820999 T936 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 821000 T936 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 821000 T936 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 821002 T936 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 821003 T936 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 821003 T936 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 821014 T936 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 821021 T936 oass.SolrIndexSearcher.<init> Opening Searcher@fffc622 main
[junit4:junit4] 2> 821022 T936 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 821022 T936 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 821026 T939 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fffc622 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 821028 T936 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 821028 T936 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49961/nikd collection:collection1 shard:shard2
[junit4:junit4] 2> 821031 T936 oasc.ZkController.register We are http://127.0.0.1:49961/nikd/collection1/ and leader is http://127.0.0.1:10744/nikd/collection1/
[junit4:junit4] 2> 821031 T936 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49961/nikd
[junit4:junit4] 2> 821031 T936 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 821032 T936 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C324 name=collection1 org.apache.solr.core.SolrCore@396eedcc url=http://127.0.0.1:49961/nikd/collection1 node=127.0.0.1:49961_nikd C324_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49961_nikd, base_url=http://127.0.0.1:49961/nikd}
[junit4:junit4] 2> 821032 T940 C324 P49961 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 821032 T936 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 821033 T940 C324 P49961 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 821033 T940 C324 P49961 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 821033 T940 C324 P49961 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 821034 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 821034 T940 C324 P49961 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 821034 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 821035 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 821051 T876 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 821336 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 821339 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:50617
[junit4:junit4] 2> 821340 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 821340 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 821341 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644
[junit4:junit4] 2> 821341 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/solr.xml
[junit4:junit4] 2> 821342 T828 oasc.CoreContainer.<init> New CoreContainer 1167370402
[junit4:junit4] 2> 821343 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/'
[junit4:junit4] 2> 821343 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/'
[junit4:junit4] 2> 821487 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 821488 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 821488 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 821488 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 821489 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 821489 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 821489 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 821490 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 821490 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 821490 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 821504 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 821505 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 821505 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 821506 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 821509 T952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e11bc1d name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 821509 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 821530 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 821538 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 821540 T954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@616f32b7 name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 821540 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 821551 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 822231 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 822232 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49961_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49961/nikd"}
[junit4:junit4] 2> 822251 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822251 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822553 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50617_nikd
[junit4:junit4] 2> 822556 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50617_nikd
[junit4:junit4] 2> 822558 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822559 T857 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822559 T954 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822558 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 822559 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 822559 T902 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822559 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 822559 T870 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822559 T886 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822559 T935 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822560 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 822560 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 822560 T918 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822560 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 822561 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 822574 T850 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 822579 T955 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 822580 T955 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 823053 T876 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 823053 T876 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:49961_nikd&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 823756 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 823758 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50617_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50617/nikd"}
[junit4:junit4] 2> 823758 T851 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 823758 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 823775 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 823775 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 824583 T955 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/collection1
[junit4:junit4] 2> 824583 T955 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 824584 T955 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 824584 T955 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 824586 T955 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/collection1/'
[junit4:junit4] 2> 824587 T955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/collection1/lib/README' to classloader
[junit4:junit4] 2> 824588 T955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 824641 T955 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 824702 T955 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 824704 T955 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 824712 T955 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C325 name=collection1 org.apache.solr.core.SolrCore@396eedcc url=http://127.0.0.1:49961/nikd/collection1 node=127.0.0.1:49961_nikd C325_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49961_nikd, base_url=http://127.0.0.1:49961/nikd}
[junit4:junit4] 2> 825055 T940 C325 P49961 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:10744/nikd/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 825055 T940 C325 P49961 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49961/nikd START replicas=[http://127.0.0.1:10744/nikd/collection1/] nUpdates=100
[junit4:junit4] 2> 825056 T940 C325 P49961 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 825056 T940 C325 P49961 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 825056 T940 C325 P49961 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 825056 T940 C325 P49961 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 825057 T940 C325 P49961 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 825057 T940 C325 P49961 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:10744/nikd/collection1/. core=collection1
[junit4:junit4] 2> 825058 T940 C325 P49961 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C326 name=collection1 org.apache.solr.core.SolrCore@503d7792 url=http://127.0.0.1:10744/nikd/collection1 node=127.0.0.1:10744_nikd C326_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:10744_nikd, base_url=http://127.0.0.1:10744/nikd, leader=true}
[junit4:junit4] 2> 825066 T877 C326 P10744 oasc.SolrCore.execute [collection1] webapp=/nikd path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 825071 T878 C326 P10744 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 825077 T878 C326 P10744 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 825077 T878 C326 P10744 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 825081 T878 C326 P10744 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 825082 T878 C326 P10744 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 825084 T878 C326 P10744 oass.SolrIndexSearcher.<init> Opening Searcher@e2a01da realtime
[junit4:junit4] 2> 825084 T878 C326 P10744 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 825084 T878 C326 P10744 oasup.LogUpdateProcessor.finish [collection1] webapp=/nikd path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 825085 T940 C325 P49961 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 825086 T940 C325 P49961 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 825088 T879 C326 P10744 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 825088 T879 C326 P10744 oasc.SolrCore.execute [collection1] webapp=/nikd path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 825089 T940 C325 P49961 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 825089 T940 C325 P49961 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 825089 T940 C325 P49961 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 825092 T879 C326 P10744 oasc.SolrCore.execute [collection1] webapp=/nikd path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 825093 T940 C325 P49961 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 825095 T940 C325 P49961 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index.20130614010101689
[junit4:junit4] 2> 825096 T940 C325 P49961 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index.20130614010101689 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7d8f2a96 fullCopy=false
[junit4:junit4] 2> 825100 T879 C326 P10744 oasc.SolrCore.execute [collection1] webapp=/nikd path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 825102 T940 C325 P49961 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 825105 T940 C325 P49961 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 825105 T940 C325 P49961 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 825111 T940 C325 P49961 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 825112 T940 C325 P49961 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 825112 T940 C325 P49961 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 825113 T940 C325 P49961 oass.SolrIndexSearcher.<init> Opening Searcher@39414c45 main
[junit4:junit4] 2> 825115 T939 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39414c45 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 825115 T940 C325 P49961 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index.20130614010101689 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index.20130614010101689;done=true>>]
[junit4:junit4] 2> 825116 T940 C325 P49961 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index.20130614010101689
[junit4:junit4] 2> 825116 T940 C325 P49961 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty5/index.20130614010101689
[junit4:junit4] 2> 825117 T940 C325 P49961 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 825117 T940 C325 P49961 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 825118 T940 C325 P49961 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 825118 T940 C325 P49961 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 825120 T940 C325 P49961 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 825280 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 825282 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49961_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49961/nikd"}
[junit4:junit4] 2> 825297 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825297 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 825384 T955 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 825397 T955 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 825399 T955 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 825411 T955 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 825424 T955 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 825440 T955 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 825442 T955 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 825442 T955 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 825442 T955 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 825443 T955 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 825443 T955 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 825444 T955 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 825444 T955 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371189657644/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty6/
[junit4:junit4] 2> 825444 T955 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ee53fc
[junit4:junit4] 2> 825445 T955 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 825446 T955 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty6
[junit4:junit4] 2> 825446 T955 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty6/index/
[junit4:junit4] 2> 825447 T955 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 825448 T955 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty6/index
[junit4:junit4] 2> 825452 T955 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 825453 T955 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 825455 T955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 825455 T955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 825456 T955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 825456 T955 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 825457 T955 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 825457 T955 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 825460 T955 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 825460 T955 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 825460 T955 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 825473 T955 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 825481 T955 oass.SolrIndexSearcher.<init> Opening Searcher@12b540c9 main
[junit4:junit4] 2> 825482 T955 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 825482 T955 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 825485 T958 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12b540c9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 825487 T955 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 825487 T955 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50617/nikd collection:collection1 shard:shard3
[junit4:junit4] 2> 825490 T955 oasc.ZkController.register We are http://127.0.0.1:50617/nikd/collection1/ and leader is http://127.0.0.1:32169/nikd/collection1/
[junit4:junit4] 2> 825490 T955 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50617/nikd
[junit4:junit4] 2> 825491 T955 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 825491 T955 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C327 name=collection1 org.apache.solr.core.SolrCore@5477ac6c url=http://127.0.0.1:50617/nikd/collection1 node=127.0.0.1:50617_nikd C327_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:50617_nikd, base_url=http://127.0.0.1:50617/nikd}
[junit4:junit4] 2> 825491 T959 C327 P50617 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 825492 T955 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 825492 T959 C327 P50617 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 825493 T959 C327 P50617 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 825493 T959 C327 P50617 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 825493 T828 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 825494 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 825494 T959 C327 P50617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 825494 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 825513 T892 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 825773 T828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 825777 T828 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:36285
[junit4:junit4] 2> 825777 T828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 825778 T828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 825778 T828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107
[junit4:junit4] 2> 825779 T828 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107/solr.xml
[junit4:junit4] 2> 825779 T828 oasc.CoreContainer.<init> New CoreContainer 621327709
[junit4:junit4] 2> 825780 T828 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107/'
[junit4:junit4] 2> 825781 T828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107/'
[junit4:junit4] 2> 825904 T828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 825905 T828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 825906 T828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 825906 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 825906 T828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 825907 T828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 825907 T828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 825908 T828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 825908 T828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 825909 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 825924 T828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 825925 T828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46037/solr
[junit4:junit4] 2> 825925 T828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 825926 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 825929 T971 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51d23c48 name:ZooKeeperConnection Watcher:127.0.0.1:46037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 825930 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 825945 T828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 825954 T828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 825955 T973 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6979800b name:ZooKeeperConnection Watcher:127.0.0.1:46037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 825956 T828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 825966 T828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 826801 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 826802 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50617_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50617/nikd"}
[junit4:junit4] 2> 826818 T973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826819 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826818 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 826969 T828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36285_nikd
[junit4:junit4] 2> 826998 T828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36285_nikd
[junit4:junit4] 2> 827001 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 827001 T973 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827001 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 827002 T954 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827002 T973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827001 T902 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827001 T935 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827001 T870 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827003 T850 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827001 T886 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827001 T857 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827003 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827003 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827003 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827002 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827002 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827004 T918 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 827004 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 827010 T974 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 827011 T974 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 827516 T892 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 827516 T892 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:50617_nikd&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 828324 T851 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 828325 T851 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36285_nikd",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36285/nikd"}
[junit4:junit4] 2> 828326 T851 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 828326 T851 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 828343 T973 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828344 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 828343 T935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 829014 T974 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107/collection1
[junit4:junit4] 2> 829014 T974 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 829015 T974 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 829015 T974 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 829017 T974 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107/collection1/'
[junit4:junit4] 2> 829018 T974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371189662107/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 829019 T974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.
[...truncated too long message...]
Solr cannot talk to ZK
[junit4:junit4] 2> 969767 T918 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 969767 T918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 12)
[junit4:junit4] 2> 969767 T918 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 969768 T918 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 970831 T992 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:38577/nikd couldn't connect to http://127.0.0.1:10744/nikd/collection1/, counting as success
[junit4:junit4] 2> 970832 T992 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38577/nikd DONE. sync succeeded
[junit4:junit4] 2> 970832 T992 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 970833 T992 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4:junit4] 2> 970833 T992 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1060e3f5
[junit4:junit4] 2> 970835 T992 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=21,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=74,cumulative_deletesById=25,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 970836 T992 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 970836 T992 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 970837 T992 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 970846 T992 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 970847 T992 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 970848 T992 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty8 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty8;done=false>>]
[junit4:junit4] 2> 970848 T992 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty8
[junit4:junit4] 2> 970849 T992 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty8/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty8/index;done=false>>]
[junit4:junit4] 2> 970849 T992 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371189629130/jetty8/index
[junit4:junit4] 2> 970850 T992 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 970850 T992 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 11)
[junit4:junit4] 2> 970850 T992 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 970851 T992 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 11)
[junit4:junit4] 2> 970851 T992 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 970852 T992 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=Lucene41(blocksize=128), text=Pulsing41(freqCutoff=18 minBlockSize=49 maxBlockSize=167), _version_=PostingsFormat(name=Lucene41WithOrds), rnd_b=PostingsFormat(name=MockSep), intDefault=PostingsFormat(name=Lucene41WithOrds), timestamp=Lucene41(blocksize=128), id=PostingsFormat(name=MockSep), a_t=Lucene41(blocksize=128), range_facet_sl=PostingsFormat(name=MockSep), range_facet_si=Lucene41(blocksize=128), other_tl1=PostingsFormat(name=Lucene41WithOrds), multiDefault=Lucene41(blocksize=128), a_si=PostingsFormat(name=Lucene41WithOrds)}, docValues:{timestamp=DocValuesFormat(name=Disk)}, sim=DefaultSimilarity, locale=th, timezone=US/Indiana-Starke
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=113103952,total=378535936
[junit4:junit4] 2> NOTE: All tests run in this JVM: [CurrencyFieldXmlFileTest, ChaosMonkeyShardSplitTest, TestStressReorder, CoreContainerCoreInitFailuresTest, URLClassifyProcessorTest, TestDistributedSearch, SolrCoreCheckLockOnStartupTest, SpellPossibilityIteratorTest, BasicZkTest, PingRequestHandlerTest, JSONWriterTest, TestHashPartitioner, UniqFieldsUpdateProcessorFactoryTest, TestClassNameShortening, ZkSolrClientTest, LeaderElectionIntegrationTest, TestCSVResponseWriter, HighlighterConfigTest, TestDistributedGrouping, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 178.93s, 1 test, 1 failure <<< FAILURES!
[...truncated 677 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:385: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:365: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 56 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4069 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4069/
2 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<210> but was:<42>
Stack Trace:
java.lang.AssertionError: expected:<210> but was:<42>
at __randomizedtesting.SeedInfo.seed([9D654810402FC7CB:1C83C6083770A7F7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:49730/hq/qf returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:49730/hq/qf returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([9D654810402FC7CB:1C83C6083770A7F7]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9207 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 162152 T237 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /rhm/w
[junit4:junit4] 2> 162158 T237 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371335366311
[junit4:junit4] 2> 162160 T237 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 162160 T238 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 162261 T237 oasc.ZkTestServer.run start zk server on port:30938
[junit4:junit4] 2> 162262 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162267 T244 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ac43d60 name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162267 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162268 T237 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 162278 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162281 T246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3a9ea0 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162281 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162281 T237 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 162293 T237 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 162296 T237 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 162299 T237 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 162303 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 162304 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 162316 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 162317 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 162421 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 162422 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 162426 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 162427 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 162435 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 162436 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 162440 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 162440 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 162444 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 162445 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 162449 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 162449 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 162453 T237 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 162454 T237 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 162689 T237 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 162695 T237 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30941
[junit4:junit4] 2> 162695 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 162696 T237 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 162696 T237 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611
[junit4:junit4] 2> 162697 T237 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/solr.xml
[junit4:junit4] 2> 162697 T237 oasc.CoreContainer.<init> New CoreContainer 2087423458
[junit4:junit4] 2> 162698 T237 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/'
[junit4:junit4] 2> 162699 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/'
[junit4:junit4] 2> 162816 T237 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 162817 T237 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 162817 T237 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 162818 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 162818 T237 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 162818 T237 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 162819 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 162819 T237 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 162820 T237 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 162820 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 162837 T237 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 162838 T237 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30938/solr
[junit4:junit4] 2> 162838 T237 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 162840 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162843 T257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2317549f name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162844 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162867 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 162881 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 162885 T259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd22554 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 162885 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 162888 T237 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 162899 T237 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 162903 T237 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 162906 T237 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30941_rhm%2Fw
[junit4:junit4] 2> 162908 T237 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30941_rhm%2Fw
[junit4:junit4] 2> 162918 T237 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 162933 T237 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 162937 T237 oasc.Overseer.start Overseer (id=89871834567016451-127.0.0.1:30941_rhm%2Fw-n_0000000000) starting
[junit4:junit4] 2> 162947 T237 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 162957 T261 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 162958 T237 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 162962 T237 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 162964 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 162968 T260 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 162972 T262 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 162972 T262 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 164473 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 164474 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30941_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30941/rhm/w"}
[junit4:junit4] 2> 164475 T260 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 164475 T260 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 164497 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 164976 T262 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1
[junit4:junit4] 2> 164976 T262 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 164977 T262 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 164977 T262 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 164979 T262 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/'
[junit4:junit4] 2> 164981 T262 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/lib/README' to classloader
[junit4:junit4] 2> 164982 T262 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 165038 T262 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 165107 T262 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 165209 T262 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 165218 T262 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 165617 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
[junit4:junit4] 2> 165885 T262 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 165899 T262 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 165903 T262 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 165931 T262 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 165936 T262 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 165941 T262 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 165943 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 165943 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 165943 T262 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 165945 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 165945 T262 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 165946 T262 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 165946 T262 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371335366611/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/
[junit4:junit4] 2> 165946 T262 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@69fba118
[junit4:junit4] 2> 165947 T262 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 165948 T262 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data
[junit4:junit4] 2> 165948 T262 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/index/
[junit4:junit4] 2> 165948 T262 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 165949 T262 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/index
[junit4:junit4] 2> 165954 T262 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@43730ad7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@504182ad; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 165955 T262 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 165958 T262 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 165958 T262 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 165959 T262 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 165960 T262 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 165960 T262 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 165960 T262 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 165961 T262 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 165961 T262 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 165962 T262 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 165977 T262 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 165984 T262 oass.SolrIndexSearcher.<init> Opening Searcher@5fa05dda main
[junit4:junit4] 2> 165985 T262 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/control/data/tlog
[junit4:junit4] 2> 165986 T262 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 165986 T262 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 165990 T263 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5fa05dda main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 165993 T262 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 165993 T262 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30941/rhm/w collection:control_collection shard:shard1
[junit4:junit4] 2> 165994 T262 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 166014 T262 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 166025 T262 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 166026 T262 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 166026 T262 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:30941/rhm/w/collection1/
[junit4:junit4] 2> 166026 T262 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 166027 T262 oasc.SyncStrategy.syncToMe http://127.0.0.1:30941/rhm/w/collection1/ has no replicas
[junit4:junit4] 2> 166027 T262 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:30941/rhm/w/collection1/
[junit4:junit4] 2> 166027 T262 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 167516 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 167536 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 167565 T262 oasc.ZkController.register We are http://127.0.0.1:30941/rhm/w/collection1/ and leader is http://127.0.0.1:30941/rhm/w/collection1/
[junit4:junit4] 2> 167565 T262 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30941/rhm/w
[junit4:junit4] 2> 167565 T262 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 167566 T262 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 167566 T262 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 167575 T262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 167576 T237 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 167577 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 167578 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 167598 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 167600 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 167602 T266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3df4fd50 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 167603 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 167605 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 167608 T237 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 167877 T237 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 167880 T237 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30946
[junit4:junit4] 2> 167881 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 167882 T237 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 167882 T237 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760
[junit4:junit4] 2> 167883 T237 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/solr.xml
[junit4:junit4] 2> 167883 T237 oasc.CoreContainer.<init> New CoreContainer 260157985
[junit4:junit4] 2> 167884 T237 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/'
[junit4:junit4] 2> 167885 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/'
[junit4:junit4] 2> 168018 T237 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 168019 T237 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 168019 T237 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 168020 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 168020 T237 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 168021 T237 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 168021 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 168022 T237 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 168022 T237 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 168023 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 168041 T237 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 168041 T237 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30938/solr
[junit4:junit4] 2> 168042 T237 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 168043 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 168054 T277 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51c5f268 name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 168055 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 168069 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 168082 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 168085 T279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a3f8ce2 name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 168085 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 168091 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 169043 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 169044 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30941_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30941/rhm/w"}
[junit4:junit4] 2> 169066 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169066 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169066 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169095 T237 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30946_rhm%2Fw
[junit4:junit4] 2> 169098 T237 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30946_rhm%2Fw
[junit4:junit4] 2> 169100 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 169101 T259 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 169101 T279 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 169101 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 169102 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 169102 T266 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 169115 T280 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 169116 T280 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 170573 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 170574 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30946_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30946/rhm/w"}
[junit4:junit4] 2> 170575 T260 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 170575 T260 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 170589 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 170589 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 170589 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 171119 T280 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1
[junit4:junit4] 2> 171119 T280 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 171120 T280 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 171120 T280 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 171122 T280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/'
[junit4:junit4] 2> 171124 T280 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 171125 T280 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/lib/README' to classloader
[junit4:junit4] 2> 171179 T280 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 171242 T280 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 171344 T280 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 171353 T280 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 171957 T280 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 171967 T280 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 171970 T280 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 171992 T280 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 171996 T280 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 172000 T280 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 172001 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 172001 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 172002 T280 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 172003 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 172003 T280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 172004 T280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 172004 T280 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371335371760/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/
[junit4:junit4] 2> 172004 T280 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@69fba118
[junit4:junit4] 2> 172005 T280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 172005 T280 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1
[junit4:junit4] 2> 172005 T280 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/index/
[junit4:junit4] 2> 172005 T280 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 172006 T280 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/index
[junit4:junit4] 2> 172009 T280 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 172009 T280 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 172012 T280 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 172012 T280 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 172012 T280 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 172013 T280 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 172013 T280 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 172014 T280 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 172014 T280 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 172015 T280 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 172015 T280 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 172026 T280 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 172031 T280 oass.SolrIndexSearcher.<init> Opening Searcher@6beb12d6 main
[junit4:junit4] 2> 172031 T280 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty1/tlog
[junit4:junit4] 2> 172032 T280 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 172032 T280 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 172036 T281 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6beb12d6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 172038 T280 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 172038 T280 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30946/rhm/w collection:collection1 shard:shard1
[junit4:junit4] 2> 172039 T280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 172058 T280 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 172062 T280 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 172063 T280 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 172063 T280 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 172063 T280 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 172064 T280 oasc.SyncStrategy.syncToMe http://127.0.0.1:30946/rhm/w/collection1/ has no replicas
[junit4:junit4] 2> 172064 T280 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 172064 T280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 172094 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 172111 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 172111 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 172111 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 172130 T280 oasc.ZkController.register We are http://127.0.0.1:30946/rhm/w/collection1/ and leader is http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 172130 T280 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30946/rhm/w
[junit4:junit4] 2> 172130 T280 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 172130 T280 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 172131 T280 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 172133 T280 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 172134 T237 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 172134 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 172135 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 172319 T237 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 172321 T237 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30949
[junit4:junit4] 2> 172322 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 172322 T237 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 172323 T237 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299
[junit4:junit4] 2> 172323 T237 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/solr.xml
[junit4:junit4] 2> 172323 T237 oasc.CoreContainer.<init> New CoreContainer 262392533
[junit4:junit4] 2> 172324 T237 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/'
[junit4:junit4] 2> 172324 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/'
[junit4:junit4] 2> 172425 T237 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 172425 T237 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 172425 T237 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 172426 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 172426 T237 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 172427 T237 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 172427 T237 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 172427 T237 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 172428 T237 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 172428 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 172440 T237 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 172441 T237 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30938/solr
[junit4:junit4] 2> 172441 T237 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 172442 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 172445 T293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@781c4dc3 name:ZooKeeperConnection Watcher:127.0.0.1:30938 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 172446 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 172449 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 172457 T237 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 172460 T295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5360b61f name:ZooKeeperConnection Watcher:127.0.0.1:30938/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 172460 T237 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 172465 T237 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 173468 T237 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30949_rhm%2Fw
[junit4:junit4] 2> 173497 T237 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30949_rhm%2Fw
[junit4:junit4] 2> 173500 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 173501 T279 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173501 T295 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173501 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173501 T259 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173502 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173503 T266 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 173510 T296 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 173510 T296 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 173625 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 173626 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30946_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30946/rhm/w"}
[junit4:junit4] 2> 173660 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 173661 T260 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 173661 T260 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 173676 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173676 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173676 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 173676 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 174518 T296 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1
[junit4:junit4] 2> 174518 T296 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 174519 T296 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 174520 T296 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 174522 T296 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/'
[junit4:junit4] 2> 174523 T296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 174524 T296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/lib/README' to classloader
[junit4:junit4] 2> 174578 T296 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 174641 T296 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 174742 T296 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 174754 T296 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 175390 T296 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 175400 T296 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 175403 T296 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 175424 T296 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 175429 T296 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 175433 T296 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 175434 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 175435 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 175435 T296 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 175436 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 175436 T296 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 175437 T296 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 175437 T296 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371335376299/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/
[junit4:junit4] 2> 175437 T296 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@69fba118
[junit4:junit4] 2> 175438 T296 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 175438 T296 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2
[junit4:junit4] 2> 175438 T296 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index/
[junit4:junit4] 2> 175439 T296 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 175439 T296 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index
[junit4:junit4] 2> 175442 T296 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 175442 T296 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 175444 T296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 175445 T296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 175445 T296 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 175446 T296 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 175446 T296 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 175447 T296 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 175447 T296 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 175447 T296 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 175448 T296 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 175458 T296 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 175464 T296 oass.SolrIndexSearcher.<init> Opening Searcher@4be04bd0 main
[junit4:junit4] 2> 175464 T296 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/tlog
[junit4:junit4] 2> 175465 T296 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 175465 T296 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 175469 T297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4be04bd0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 175471 T296 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 175471 T296 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30949/rhm/w collection:collection1 shard:shard1
[junit4:junit4] 2> 175500 T296 oasc.ZkController.register We are http://127.0.0.1:30949/rhm/w/collection1/ and leader is http://127.0.0.1:30946/rhm/w/collection1/
[junit4:junit4] 2> 175500 T296 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30949/rhm/w
[junit4:junit4] 2> 175501 T296 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 175501 T296 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C109 name=collection1 org.apache.solr.core.SolrCore@a41ba9d url=http://127.0.0.1:30949/rhm/w/collection1 node=127.0.0.1:30949_rhm%2Fw C109_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:30949_rhm%2Fw, base_url=http://127.0.0.1:30949/rhm/w}
[junit4:junit4] 2> 175502 T298 C109 P30949 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 175502 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 175502 T296 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 175502 T298 C109 P30949 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 175503 T298 C109 P30949 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 175503 T237 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 175504 T237 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 175504 T298 C109 P30949 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 175504 T237 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 175519 T237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 175520 T269 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C110 name=collection1 org.apache.solr.core.SolrCore@33287c71 url=http://127.0.0.1:30941/rhm/w/collection1 node=127.0.0.1:30941_rhm%2Fw C110_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:30941_rhm%2Fw, base_url=http://127.0.0.1:30941/rhm/w, leader=true}
[junit4:junit4] 2> 175532 T250 C110 P30941 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@43730ad7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@504182ad; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 175532 T250 C110 P30941 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 175537 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[1 (1437949367085432832)]} 0 8
[junit4:junit4] 2> 175539 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10001 (1437949367087529984)]} 0 7
[junit4:junit4] 2> ASYNC NEW_CORE C111 name=collection1 org.apache.solr.core.SolrCore@5e1f92d8 url=http://127.0.0.1:30946/rhm/w/collection1 node=127.0.0.1:30946_rhm%2Fw C111_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:30946_rhm%2Fw, base_url=http://127.0.0.1:30946/rhm/w, leader=true}
[junit4:junit4] 2> 175556 T271 C111 P30946 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 175556 T271 C111 P30946 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 175560 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10001 (1437949367109550080)]} 0 7
[junit4:junit4] 2> 175561 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[1 (1437949367112695808)]} 0 6
[junit4:junit4] 2> 175657 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10002 (1437949367216504832)]} 0 3
[junit4:junit4] 2> 175658 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[2 (1437949367218601984)]} 0 2
[junit4:junit4] 2> 175662 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10002 (1437949367221747712)]} 0 2
[junit4:junit4] 2> 175663 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[2 (1437949367222796288)]} 0 2
[junit4:junit4] 2> 175706 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10003 (1437949367267885056)]} 0 2
[junit4:junit4] 2> 175708 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[3 (1437949367269982208)]} 0 3
[junit4:junit4] 2> 175711 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10003 (1437949367274176512)]} 0 2
[junit4:junit4] 2> 175714 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[3 (1437949367276273664)]} 0 3
[junit4:junit4] 2> 175805 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10001 (-1437949367373791232)]} 0 1
[junit4:junit4] 2> 175809 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[1 (-1437949367377985536)]} 0 1
[junit4:junit4] 2> 175809 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10001 (-1437949367377985536)]} 0 1
[junit4:junit4] 2> 175813 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[1 (-1437949367381131264)]} 0 1
[junit4:junit4] 2> 175814 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10004 (1437949367382179840)]} 0 2
[junit4:junit4] 2> 175817 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[4 (1437949367384276992)]} 0 2
[junit4:junit4] 2> 175819 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10004 (1437949367387422720)]} 0 2
[junit4:junit4] 2> 175822 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[4 (1437949367389519872)]} 0 3
[junit4:junit4] 2> 175827 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10005 (1437949367394762752)]} 0 3
[junit4:junit4] 2> 175830 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[5 (1437949367398957056)]} 0 2
[junit4:junit4] 2> 175833 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10005 (1437949367401054208)]} 0 3
[junit4:junit4] 2> 175835 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[5 (1437949367403151360)]} 0 2
[junit4:junit4] 2> 175922 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10006 (1437949367494377472)]} 0 2
[junit4:junit4] 2> 175923 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[6 (1437949367495426048)]} 0 3
[junit4:junit4] 2> 175927 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10006 (1437949367500668928)]} 0 2
[junit4:junit4] 2> 175929 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[6 (1437949367501717504)]} 0 3
[junit4:junit4] 2> 175948 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10002 (-1437949367523737600)]} 0 1
[junit4:junit4] 2> 175950 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[2 (-1437949367525834752)]} 0 1
[junit4:junit4] 2> 175953 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10002 (-1437949367528980480)]} 0 1
[junit4:junit4] 2> 175954 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[2 (-1437949367528980481)]} 0 1
[junit4:junit4] 2> 175959 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[7 (1437949367533174784)]} 0 3
[junit4:junit4] 2> 175960 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10007 (1437949367533174785)]} 0 3
[junit4:junit4] 2> 175965 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[7 (1437949367539466240)]} 0 3
[junit4:junit4] 2> 175967 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10007 (1437949367541563392)]} 0 3
[junit4:junit4] 2> 176058 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[4 (-1437949367638032384)]} 0 1
[junit4:junit4] 2> 176059 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10004 (-1437949367640129536)]} 0 1
[junit4:junit4] 2> 176062 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[4 (-1437949367643275264)]} 0 1
[junit4:junit4] 2> 176063 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10004 (-1437949367643275265)]} 0 1
[junit4:junit4] 2> 176068 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[8 (1437949367647469568)]} 0 3
[junit4:junit4] 2> 176069 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10008 (1437949367648518144)]} 0 3
[junit4:junit4] 2> 176074 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[8 (1437949367653761024)]} 0 3
[junit4:junit4] 2> 176075 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10008 (1437949367654809600)]} 0 3
[junit4:junit4] 2> 176102 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[9 (1437949367684169728)]} 0 2
[junit4:junit4] 2> 176104 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10009 (1437949367686266880)]} 0 2
[junit4:junit4] 2> 176107 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[9 (1437949367689412608)]} 0 2
[junit4:junit4] 2> 176109 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10009 (1437949367691509760)]} 0 2
[junit4:junit4] 2> 176138 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10 (1437949367720869888)]} 0 3
[junit4:junit4] 2> 176139 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10010 (1437949367722967040)]} 0 2
[junit4:junit4] 2> 176143 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10 (1437949367727161344)]} 0 2
[junit4:junit4] 2> 176144 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10010 (1437949367728209920)]} 0 2
[junit4:junit4] 2> 176147 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[6 (-1437949367732404224)]} 0 1
[junit4:junit4] 2> 176148 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10006 (-1437949367733452800)]} 0 1
[junit4:junit4] 2> 176151 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[6 (-1437949367735549952)]} 0 1
[junit4:junit4] 2> 176151 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10006 (-1437949367736598528)]} 0 0
[junit4:junit4] 2> 176155 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[11 (1437949367738695680)]} 0 2
[junit4:junit4] 2> 176156 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10011 (1437949367739744256)]} 0 2
[junit4:junit4] 2> 176160 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[11 (1437949367743938560)]} 0 2
[junit4:junit4] 2> 176161 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10011 (1437949367744987136)]} 0 2
[junit4:junit4] 2> 176188 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[7 (-1437949367774347264)]} 0 1
[junit4:junit4] 2> 176189 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10007 (-1437949367775395840)]} 0 1
[junit4:junit4] 2> 176190 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[7 (-1437949367777492992)]} 0 0
[junit4:junit4] 2> 176191 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10007 (-1437949367778541568)]} 0 0
[junit4:junit4] 2> 176195 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[12 (1437949367780638720)]} 0 2
[junit4:junit4] 2> 176196 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10012 (1437949367781687296)]} 0 2
[junit4:junit4] 2> 176198 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[12 (1437949367784833024)]} 0 1
[junit4:junit4] 2> 176200 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10012 (1437949367785881600)]} 0 2
[junit4:junit4] 2> 176256 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[13 (1437949367845650432)]} 0 2
[junit4:junit4] 2> 176257 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10013 (1437949367845650433)]} 0 2
[junit4:junit4] 2> 176261 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10013 (1437949367849844736)]} 0 2
[junit4:junit4] 2> 176261 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[13 (1437949367849844737)]} 0 2
[junit4:junit4] 2> 176340 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10014 (1437949367933730816)]} 0 2
[junit4:junit4] 2> 176341 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[14 (1437949367934779392)]} 0 2
[junit4:junit4] 2> 176344 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[14 (1437949367937925120)]} 0 1
[junit4:junit4] 2> 176345 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10014 (1437949367938973696)]} 0 2
[junit4:junit4] 2> 176427 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[15 (1437949368023908352)]} 0 2
[junit4:junit4] 2> 176428 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10015 (1437949368024956928)]} 0 3
[junit4:junit4] 2> 176433 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[15 (1437949368030199808)]} 0 3
[junit4:junit4] 2> 176434 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10015 (1437949368031248384)]} 0 2
[junit4:junit4] 2> 176535 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[16 (1437949368137154560)]} 0 3
[junit4:junit4] 2> 176537 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10016 (1437949368138203136)]} 0 4
[junit4:junit4] 2> 176541 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[16 (1437949368144494592)]} 0 2
[junit4:junit4] 2> 176543 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10016 (1437949368145543168)]} 0 3
[junit4:junit4] 2> 176619 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[9 (-1437949368227332096)]} 0 1
[junit4:junit4] 2> 176620 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10009 (-1437949368228380672)]} 0 1
[junit4:junit4] 2> 176623 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[9 (-1437949368231526400)]} 0 1
[junit4:junit4] 2> 176624 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10009 (-1437949368231526401)]} 0 1
[junit4:junit4] 2> 176628 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[17 (1437949368235720704)]} 0 2
[junit4:junit4] 2> 176630 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10017 (1437949368236769280)]} 0 3
[junit4:junit4] 2> 176633 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[17 (1437949368240963584)]} 0 2
[junit4:junit4] 2> 176636 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10017 (1437949368243060736)]} 0 3
[junit4:junit4] 2> 176648 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[15 (-1437949368257740800)]} 0 1
[junit4:junit4] 2> 176651 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10015 (-1437949368259837952)]} 0 1
[junit4:junit4] 2> 176652 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[15 (-1437949368261935104)]} 0 1
[junit4:junit4] 2> 176655 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10015 (-1437949368265080832)]} 0 1
[junit4:junit4] 2> 176658 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[18 (1437949368266129408)]} 0 3
[junit4:junit4] 2> 176660 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10018 (1437949368269275136)]} 0 2
[junit4:junit4] 2> 176663 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[18 (1437949368272420864)]} 0 2
[junit4:junit4] 2> 176667 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10018 (1437949368275566592)]} 0 3
[junit4:junit4] 2> 176694 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 176695 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 176724 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[19 (1437949368335335424)]} 0 3
[junit4:junit4] 2> 176729 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10019 (1437949368339529728)]} 0 3
[junit4:junit4] 2> 176729 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[19 (1437949368341626880)]} 0 2
[junit4:junit4] 2> 176733 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176733 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176733 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176733 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 176735 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10019 (1437949368346869760)]} 0 3
[junit4:junit4] 2> 176789 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[17 (-1437949368405590016)]} 0 1
[junit4:junit4] 2> 176796 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10017 (-1437949368411881472)]} 0 1
[junit4:junit4] 2> C109_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:30949_rhm%2Fw, base_url=http://127.0.0.1:30949/rhm/w}
[junit4:junit4] 2> 176806 T285 C109 P30949 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 176807 T285 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 176809 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[17 (-1437949368409784320)]} 0 5
[junit4:junit4] 2> 176810 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10017 (-1437949368416075776)]} 0 1
[junit4:junit4] 2> 176810 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[17 (-1437949368409784320)]} 0 18
[junit4:junit4] 2> 176811 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10017 (-1437949368416075776)]} 0 12
[junit4:junit4] 2> 176816 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[20 (1437949368432852992)]} 0 2
[junit4:junit4] 2> 176818 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10020 (1437949368432852993)]} 0 4
[junit4:junit4] 2> 176827 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1437949368438095872)]} 0 3
[junit4:junit4] 2> 176828 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[20 (1437949368438095872)]} 0 9
[junit4:junit4] 2> 176829 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1437949368440193024)]} 0 4
[junit4:junit4] 2> 176829 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10020 (1437949368440193024)]} 0 8
[junit4:junit4] 2> 176924 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[21 (1437949368546099200)]} 0 2
[junit4:junit4] 2> 176925 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10021 (1437949368547147776)]} 0 2
[junit4:junit4] 2> 176933 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1437949368551342080)]} 0 2
[junit4:junit4] 2> 176934 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[21 (1437949368551342080)]} 0 7
[junit4:junit4] 2> 176935 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1437949368552390656)]} 0 2
[junit4:junit4] 2> 176936 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10021 (1437949368552390656)]} 0 8
[junit4:junit4] 2> 176976 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[22 (1437949368600625152)]} 0 2
[junit4:junit4] 2> 176978 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10022 (1437949368602722304)]} 0 2
[junit4:junit4] 2> 176987 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1437949368606916608)]} 0 2
[junit4:junit4] 2> 176988 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[22 (1437949368606916608)]} 0 8
[junit4:junit4] 2> 176989 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1437949368607965184)]} 0 3
[junit4:junit4] 2> 176990 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10022 (1437949368607965184)]} 0 9
[junit4:junit4] 2> 177074 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[23 (1437949368702337024)]} 0 2
[junit4:junit4] 2> 177075 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10023 (1437949368704434176)]} 0 2
[junit4:junit4] 2> 177084 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1437949368708628480)]} 0 2
[junit4:junit4] 2> 177085 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[23 (1437949368708628480)]} 0 8
[junit4:junit4] 2> 177086 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1437949368710725632)]} 0 2
[junit4:junit4] 2> 177087 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10023 (1437949368710725632)]} 0 8
[junit4:junit4] 2> 177149 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[24 (1437949368780980224)]} 0 3
[junit4:junit4] 2> 177150 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10024 (1437949368783077376)]} 0 2
[junit4:junit4] 2> 177159 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1437949368787271680)]} 0 2
[junit4:junit4] 2> 177160 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[24 (1437949368787271680)]} 0 8
[junit4:junit4] 2> 177160 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1437949368788320256)]} 0 2
[junit4:junit4] 2> 177161 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10024 (1437949368788320256)]} 0 8
[junit4:junit4] 2> 177232 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[22 (-1437949368870109184)]} 0 1
[junit4:junit4] 2> 177233 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10022 (-1437949368871157760)]} 0 0
[junit4:junit4] 2> 177239 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1437949368874303488)]} 0 1
[junit4:junit4] 2> 177240 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[22 (-1437949368874303488)]} 0 5
[junit4:junit4] 2> 177240 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1437949368875352064)]} 0 1
[junit4:junit4] 2> 177241 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10022 (-1437949368875352064)]} 0 5
[junit4:junit4] 2> 177246 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[25 (1437949368882692096)]} 0 3
[junit4:junit4] 2> 177247 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10025 (1437949368883740672)]} 0 3
[junit4:junit4] 2> 177256 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1437949368888983552)]} 0 2
[junit4:junit4] 2> 177257 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[25 (1437949368888983552)]} 0 8
[junit4:junit4] 2> 177257 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1437949368890032128)]} 0 2
[junit4:junit4] 2> 177259 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10025 (1437949368890032128)]} 0 9
[junit4:junit4] 2> 177268 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[26 (1437949368906809344)]} 0 2
[junit4:junit4] 2> 177269 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10026 (1437949368907857920)]} 0 2
[junit4:junit4] 2> 177279 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1437949368913100800)]} 0 2
[junit4:junit4] 2> 177280 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[26 (1437949368913100800)]} 0 8
[junit4:junit4] 2> 177280 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1437949368913100801)]} 0 2
[junit4:junit4] 2> 177281 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10026 (1437949368913100801)]} 0 9
[junit4:junit4] 2> 177383 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[23 (-1437949369028444160)]} 0 1
[junit4:junit4] 2> 177384 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10023 (-1437949369029492736)]} 0 0
[junit4:junit4] 2> 177390 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[23 (-1437949369032638464)]} 0 1
[junit4:junit4] 2> 177391 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[23 (-1437949369032638464)]} 0 5
[junit4:junit4] 2> 177391 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10023 (-1437949369033687040)]} 0 1
[junit4:junit4] 2> 177392 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10023 (-1437949369033687040)]} 0 5
[junit4:junit4] 2> 177396 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[27 (1437949369041027072)]} 0 2
[junit4:junit4] 2> 177397 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10027 (1437949369042075648)]} 0 2
[junit4:junit4] 2> 177405 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1437949369046269952)]} 0 2
[junit4:junit4] 2> 177406 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[27 (1437949369046269952)]} 0 7
[junit4:junit4] 2> 177406 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1437949369047318528)]} 0 2
[junit4:junit4] 2> 177407 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10027 (1437949369047318528)]} 0 7
[junit4:junit4] 2> 177482 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[24 (-1437949369132253184)]} 0 1
[junit4:junit4] 2> 177483 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10024 (-1437949369133301760)]} 0 1
[junit4:junit4] 2> 177489 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[24 (-1437949369135398912)]} 0 1
[junit4:junit4] 2> 177490 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[24 (-1437949369135398912)]} 0 5
[junit4:junit4] 2> 177490 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10024 (-1437949369136447488)]} 0 1
[junit4:junit4] 2> 177491 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10024 (-1437949369136447488)]} 0 5
[junit4:junit4] 2> 177495 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[28 (1437949369144836096)]} 0 2
[junit4:junit4] 2> 177496 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10028 (1437949369145884672)]} 0 2
[junit4:junit4] 2> 177504 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1437949369150078976)]} 0 2
[junit4:junit4] 2> 177505 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[28 (1437949369150078976)]} 0 7
[junit4:junit4] 2> 177505 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1437949369151127552)]} 0 2
[junit4:junit4] 2> 177506 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10028 (1437949369151127552)]} 0 7
[junit4:junit4] 2> 177522 T269 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 177522 T237 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 30949
[junit4:junit4] 2> 177522 T269 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:30949_rhm%252Fw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 177523 T237 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=262392533
[junit4:junit4] 2> 177532 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[29 (1437949369183633408)]} 0 2
[junit4:junit4] 2> 177533 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10029 (1437949369184681984)]} 0 2
[junit4:junit4] 2> 177541 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1437949369188876288)]} 0 2
[junit4:junit4] 2> 177543 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1437949369189924864)]} 0 3
[junit4:junit4] 2> 177544 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10029 (1437949369189924864)]} 0 8
[junit4:junit4] 2> 177545 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[29 (1437949369188876288)]} 0 10
[junit4:junit4] 2> 177610 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10025 (-1437949369265422336)]} 0 0
[junit4:junit4] 2> 177610 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[25 (-1437949369266470912)]} 0 0
[junit4:junit4] 2> 177618 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10025 (-1437949369271713792)]} 0 0
[junit4:junit4] 2> 177618 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[25 (-1437949369270665216)]} 0 0
[junit4:junit4] 2> 177619 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10025 (-1437949369271713792)]} 0 5
[junit4:junit4] 2> 177620 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[25 (-1437949369270665216)]} 0 7
[junit4:junit4] 2> 177624 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10030 (1437949369280102400)]} 0 2
[junit4:junit4] 2> 177626 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[30 (1437949369281150976)]} 0 3
[junit4:junit4] 2> 177635 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1437949369286393856)]} 0 2
[junit4:junit4] 2> 177636 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10030 (1437949369286393856)]} 0 8
[junit4:junit4] 2> 177637 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1437949369287442432)]} 0 2
[junit4:junit4] 2> 177638 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[30 (1437949369287442432)]} 0 9
[junit4:junit4] 2> 177680 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10027 (-1437949369338822656)]} 0 1
[junit4:junit4] 2> 177681 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[27 (-1437949369340919808)]} 0 1
[junit4:junit4] 2> 177688 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[27 (-1437949369345114112)]} 0 1
[junit4:junit4] 2> 177688 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10027 (-1437949369344065536)]} 0 1
[junit4:junit4] 2> 177689 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[27 (-1437949369345114112)]} 0 5
[junit4:junit4] 2> 177689 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10027 (-1437949369344065536)]} 0 5
[junit4:junit4] 2> 177694 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[31 (1437949369353502720)]} 0 2
[junit4:junit4] 2> 177695 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10031 (1437949369354551296)]} 0 2
[junit4:junit4] 2> 177705 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1437949369359794176)]} 0 1
[junit4:junit4] 2> 177706 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[31 (1437949369359794176)]} 0 8
[junit4:junit4] 2> 177706 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1437949369360842752)]} 0 2
[junit4:junit4] 2> 177708 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10031 (1437949369360842752)]} 0 9
[junit4:junit4] 2> 177812 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[32 (1437949369476186112)]} 0 2
[junit4:junit4] 2> 177814 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10032 (1437949369478283264)]} 0 3
[junit4:junit4] 2> 177822 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1437949369482477568)]} 0 2
[junit4:junit4] 2> 177823 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1437949369484574720)]} 0 2
[junit4:junit4] 2> 177824 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[32 (1437949369482477568)]} 0 9
[junit4:junit4] 2> 177824 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10032 (1437949369484574720)]} 0 7
[junit4:junit4] 2> 177856 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10033 (1437949369522323456)]} 0 3
[junit4:junit4] 2> 177857 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[33 (1437949369522323457)]} 0 4
[junit4:junit4] 2> 177866 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1437949369529663488)]} 0 2
[junit4:junit4] 2> 177867 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[33 (1437949369529663488)]} 0 7
[junit4:junit4] 2> 177867 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1437949369528614912)]} 0 3
[junit4:junit4] 2> 177868 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10033 (1437949369528614912)]} 0 9
[junit4:junit4] 2> 177931 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[34 (1437949369602015232)]} 0 2
[junit4:junit4] 2> 177932 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10034 (1437949369603063808)]} 0 2
[junit4:junit4] 2> 177940 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1437949369607258112)]} 0 2
[junit4:junit4] 2> 177941 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[34 (1437949369607258112)]} 0 7
[junit4:junit4] 2> 177941 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1437949369608306688)]} 0 2
[junit4:junit4] 2> 177942 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10034 (1437949369608306688)]} 0 7
[junit4:junit4] 2> 178011 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[35 (1437949369685901312)]} 0 2
[junit4:junit4] 2> 178012 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10035 (1437949369686949888)]} 0 2
[junit4:junit4] 2> 178020 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1437949369691144192)]} 0 2
[junit4:junit4] 2> 178021 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[35 (1437949369691144192)]} 0 7
[junit4:junit4] 2> 178021 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1437949369692192768)]} 0 2
[junit4:junit4] 2> 178022 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10035 (1437949369692192768)]} 0 7
[junit4:junit4] 2> 178061 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[36 (1437949369737281536)]} 0 2
[junit4:junit4] 2> 178062 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10036 (1437949369738330112)]} 0 3
[junit4:junit4] 2> 178070 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1437949369743572992)]} 0 2
[junit4:junit4] 2> 178071 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[36 (1437949369743572992)]} 0 7
[junit4:junit4] 2> 178071 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1437949369744621568)]} 0 2
[junit4:junit4] 2> 178072 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10036 (1437949369744621568)]} 0 7
[junit4:junit4] 2> 178133 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[33 (-1437949369813827584)]} 0 1
[junit4:junit4] 2> 178134 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10033 (-1437949369814876160)]} 0 1
[junit4:junit4] 2> 178140 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[33 (-1437949369818021888)]} 0 1
[junit4:junit4] 2> 178140 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[33 (-1437949369818021888)]} 0 4
[junit4:junit4] 2> 178141 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10033 (-1437949369819070464)]} 0 1
[junit4:junit4] 2> 178142 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10033 (-1437949369819070464)]} 0 5
[junit4:junit4] 2> 178146 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[37 (1437949369827459072)]} 0 2
[junit4:junit4] 2> 178147 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10037 (1437949369828507648)]} 0 2
[junit4:junit4] 2> 178155 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1437949369832701952)]} 0 2
[junit4:junit4] 2> 178156 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[37 (1437949369832701952)]} 0 7
[junit4:junit4] 2> 178156 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1437949369833750528)]} 0 2
[junit4:junit4] 2> 178157 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10037 (1437949369833750528)]} 0 7
[junit4:junit4] 2> 178223 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[36 (-1437949369908199424)]} 0 1
[junit4:junit4] 2> 178224 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10036 (-1437949369909248000)]} 0 1
[junit4:junit4] 2> 178230 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[36 (-1437949369912393728)]} 0 1
[junit4:junit4] 2> 178231 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[36 (-1437949369912393728)]} 0 4
[junit4:junit4] 2> 178231 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10036 (-1437949369914490880)]} 0 1
[junit4:junit4] 2> 178232 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10036 (-1437949369914490880)]} 0 5
[junit4:junit4] 2> 178236 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[38 (1437949369921830912)]} 0 2
[junit4:junit4] 2> 178237 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10038 (1437949369922879488)]} 0 2
[junit4:junit4] 2> 178239 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 178240 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 178245 T285 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1437949369927073792)]} 0 2
[junit4:junit4] 2> 178246 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[38 (1437949369927073792)]} 0 7
[junit4:junit4] 2> 178246 T286 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={distrib.from=http://127.0.0.1:30946/rhm/w/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1437949369928122368)]} 0 2
[junit4:junit4] 2> 178247 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10038 (1437949369928122368)]} 0 7
[junit4:junit4] 2> 178282 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178282 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178282 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178282 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 178326 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[39 (1437949370016202752)]} 0 2
[junit4:junit4] 2> 178327 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10039 (1437949370017251328)]} 0 2
[junit4:junit4] 2> 178331 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[39 (1437949370021445632)]} 0 2
[junit4:junit4] 2> 178332 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10039 (1437949370022494208)]} 0 2
[junit4:junit4] 2> 178435 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[40 (1437949370130497536)]} 0 2
[junit4:junit4] 2> 178436 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10040 (1437949370131546112)]} 0 2
[junit4:junit4] 2> 178440 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[40 (1437949370135740416)]} 0 2
[junit4:junit4] 2> 178441 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10040 (1437949370136788992)]} 0 2
[junit4:junit4] 2> 178487 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[41 (1437949370185023488)]} 0 2
[junit4:junit4] 2> 178488 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10041 (1437949370185023489)]} 0 2
[junit4:junit4] 2> 178493 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[41 (1437949370191314944)]} 0 2
[junit4:junit4] 2> 178494 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10041 (1437949370192363520)]} 0 2
[junit4:junit4] 2> 178585 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[42 (1437949370286735360)]} 0 2
[junit4:junit4] 2> 178586 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10042 (1437949370287783936)]} 0 2
[junit4:junit4] 2> 178590 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[42 (1437949370293026816)]} 0 2
[junit4:junit4] 2> 178591 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10042 (1437949370294075392)]} 0 2
[junit4:junit4] 2> 178659 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10041 (-1437949370366427137)]} 0 1
[junit4:junit4] 2> 178659 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[41 (-1437949370366427136)]} 0 1
[junit4:junit4] 2> 178663 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10041 (-1437949370369572864)]} 0 1
[junit4:junit4] 2> 178663 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[41 (-1437949370370621440)]} 0 1
[junit4:junit4] 2> 178668 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10043 (1437949370374815744)]} 0 2
[junit4:junit4] 2> 178669 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[43 (1437949370374815745)]} 0 3
[junit4:junit4] 2> 178673 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10043 (1437949370380058624)]} 0 2
[junit4:junit4] 2> 178674 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[43 (1437949370381107200)]} 0 2
[junit4:junit4] 2> 178764 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10044 (1437949370475479040)]} 0 1
[junit4:junit4] 2> 178765 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[44 (1437949370476527616)]} 0 2
[junit4:junit4] 2> 178770 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10044 (1437949370481770496)]} 0 2
[junit4:junit4] 2> 178771 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[44 (1437949370481770497)]} 0 3
[junit4:junit4] 2> 178865 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10045 (1437949370580336640)]} 0 2
[junit4:junit4] 2> 178866 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[45 (1437949370581385216)]} 0 3
[junit4:junit4] 2> 178870 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10045 (1437949370586628096)]} 0 2
[junit4:junit4] 2> 178871 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[45 (1437949370587676672)]} 0 2
[junit4:junit4] 2> 178874 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10042 (-1437949370590822400)]} 0 1
[junit4:junit4] 2> 178875 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[42 (-1437949370591870976)]} 0 1
[junit4:junit4] 2> 178878 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10042 (-1437949370595016704)]} 0 1
[junit4:junit4] 2> 178879 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[42 (-1437949370596065280)]} 0 1
[junit4:junit4] 2> 178883 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10046 (1437949370600259584)]} 0 2
[junit4:junit4] 2> 178884 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[46 (1437949370601308160)]} 0 2
[junit4:junit4] 2> 178888 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10046 (1437949370605502464)]} 0 2
[junit4:junit4] 2> 178889 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[46 (1437949370606551040)]} 0 2
[junit4:junit4] 2> 178954 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10045 (-1437949370675757056)]} 0 1
[junit4:junit4] 2> 178955 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[45 (-1437949370675757057)]} 0 1
[junit4:junit4] 2> 178958 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10045 (-1437949370678902784)]} 0 1
[junit4:junit4] 2> 178959 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[45 (-1437949370679951360)]} 0 1
[junit4:junit4] 2> 178963 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10047 (1437949370684145664)]} 0 2
[junit4:junit4] 2> 178964 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[47 (1437949370685194240)]} 0 2
[junit4:junit4] 2> 178968 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10047 (1437949370689388544)]} 0 2
[junit4:junit4] 2> 178969 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[47 (1437949370690437120)]} 0 2
[junit4:junit4] 2> 179018 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10046 (-1437949370742865920)]} 0 0
[junit4:junit4] 2> 179019 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[46 (-1437949370742865921)]} 0 1
[junit4:junit4] 2> 179022 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10046 (-1437949370746011648)]} 0 1
[junit4:junit4] 2> 179023 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[46 (-1437949370747060224)]} 0 1
[junit4:junit4] 2> 179027 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10048 (1437949370751254528)]} 0 2
[junit4:junit4] 2> 179028 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[48 (1437949370752303104)]} 0 2
[junit4:junit4] 2> 179032 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10048 (1437949370756497408)]} 0 2
[junit4:junit4] 2> 179033 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[48 (1437949370757545984)]} 0 2
[junit4:junit4] 2> 179134 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10049 (1437949370862403584)]} 0 2
[junit4:junit4] 2> 179135 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[49 (1437949370862403585)]} 0 3
[junit4:junit4] 2> 179139 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10049 (1437949370868695040)]} 0 2
[junit4:junit4] 2> 179140 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[49 (1437949370869743616)]} 0 2
[junit4:junit4] 2> 179157 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10050 (1437949370887569408)]} 0 2
[junit4:junit4] 2> 179158 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[50 (1437949370888617984)]} 0 2
[junit4:junit4] 2> 179162 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10050 (1437949370892812288)]} 0 2
[junit4:junit4] 2> 179163 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[50 (1437949370893860864)]} 0 2
[junit4:junit4] 2> 179255 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10047 (-1437949370991378432)]} 0 1
[junit4:junit4] 2> 179259 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10047 (-1437949370995572736)]} 0 1
[junit4:junit4] 2> 179264 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10051 (1437949370999767040)]} 0 2
[junit4:junit4] 2> 179269 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10051 (1437949371005009920)]} 0 2
[junit4:junit4] 2> 179270 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[47 (-1437949371006058496)]} 0 1
[junit4:junit4] 2> 179274 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[47 (-1437949371010252800)]} 0 1
[junit4:junit4] 2> 179279 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[51 (1437949371015495680)]} 0 2
[junit4:junit4] 2> 179284 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[51 (1437949371020738560)]} 0 2
[junit4:junit4] 2> 179324 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10048 (-1437949371063730176)]} 0 0
[junit4:junit4] 2> 179329 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10048 (-1437949371067924480)]} 0 1
[junit4:junit4] 2> 179334 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10052 (1437949371073167360)]} 0 2
[junit4:junit4] 2> 179339 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10052 (1437949371078410240)]} 0 2
[junit4:junit4] 2> 179340 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[48 (-1437949371080507392)]} 0 1
[junit4:junit4] 2> 179344 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[48 (-1437949371083653120)]} 0 1
[junit4:junit4] 2> 179349 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[52 (1437949371088896000)]} 0 2
[junit4:junit4] 2> 179354 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[52 (1437949371094138880)]} 0 2
[junit4:junit4] 2> 179390 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10049 (-1437949371132936192)]} 0 1
[junit4:junit4] 2> 179394 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10049 (-1437949371136081920)]} 0 1
[junit4:junit4] 2> 179399 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10053 (1437949371141324800)]} 0 2
[junit4:junit4] 2> 179404 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10053 (1437949371146567680)]} 0 2
[junit4:junit4] 2> 179405 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[49 (-1437949371147616256)]} 0 1
[junit4:junit4] 2> 179409 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[49 (-1437949371151810560)]} 0 1
[junit4:junit4] 2> 179414 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[53 (1437949371157053440)]} 0 2
[junit4:junit4] 2> 179417 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10054 (1437949371160199168)]} 0 2
[junit4:junit4] 2> 179419 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[53 (1437949371162296320)]} 0 2
[junit4:junit4] 2> 179422 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10054 (1437949371165442048)]} 0 2
[junit4:junit4] 2> 179433 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[54 (1437949371176976384)]} 0 2
[junit4:junit4] 2> 179437 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10050 (-1437949371182219264)]} 0 1
[junit4:junit4] 2> 179438 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[54 (1437949371182219264)]} 0 2
[junit4:junit4] 2> 179441 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10050 (-1437949371186413568)]} 0 1
[junit4:junit4] 2> 179446 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10055 (1437949371190607872)]} 0 2
[junit4:junit4] 2> 179452 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10055 (1437949371195850752)]} 0 2
[junit4:junit4] 2> 179453 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[50 (-1437949371198996480)]} 0 0
[junit4:junit4] 2> 179457 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[50 (-1437949371203190784)]} 0 1
[junit4:junit4] 2> 179462 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[55 (1437949371207385088)]} 0 2
[junit4:junit4] 2> 179467 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[55 (1437949371212627968)]} 0 2
[junit4:junit4] 2> C109_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:30949_rhm%2Fw, base_url=http://127.0.0.1:30949/rhm/w}
[junit4:junit4] 2> 179524 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:30946/rhm/w/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 179524 T298 C109 P30949 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:30949/rhm/w START replicas=[http://127.0.0.1:30946/rhm/w/collection1/] nUpdates=100
[junit4:junit4] 2> 179525 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10056 (1437949371273445376)]} 0 2
[junit4:junit4] 2> 179527 T237 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 179527 T237 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 179530 T237 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=3core=collection1
[junit4:junit4] 2> 179531 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10056 (1437949371279736832)]} 0 2
[junit4:junit4] 2> 179532 T272 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 179531 T298 C109 P30949 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 179532 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 179533 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 179533 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 179533 T298 C109 P30949 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 179534 T298 C109 P30949 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:30946/rhm/w/collection1/. core=collection1
[junit4:junit4] 2> 179534 T298 C109 P30949 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 179539 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[56 (1437949371288125440)]} 0 2
[junit4:junit4] 2> 179544 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[56 (1437949371293368320)]} 0 2
[junit4:junit4] 2> 179547 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10052 (-1437949371297562624)]} 0 1
[junit4:junit4] 2> 179549 T273 C111 P30946 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 179552 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10052 (-1437949371300708352)]} 0 2
[junit4:junit4] 2> 179557 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10057 (1437949371306999808)]} 0 2
[junit4:junit4] 2> 179559 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[52 (-1437949371309096960)]} 0 1
[junit4:junit4] 2> 179562 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10057 (1437949371312242688)]} 0 2
[junit4:junit4] 2> 179563 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[52 (-1437949371313291264)]} 0 1
[junit4:junit4] 2> 179568 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[57 (1437949371318534144)]} 0 2
[junit4:junit4] 2> 179573 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[57 (1437949371323777024)]} 0 2
[junit4:junit4] 2> 179618 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10058 (1437949371370962944)]} 0 2
[junit4:junit4] 2> 179624 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10058 (1437949371376205824)]} 0 2
[junit4:junit4] 2> 179628 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[58 (1437949371381448704)]} 0 2
[junit4:junit4] 2> 179633 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[58 (1437949371386691584)]} 0 2
[junit4:junit4] 2> 179650 T273 C111 P30946 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@11629b3b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c982419; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 179651 T273 C111 P30946 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 179713 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10054 (-1437949371471626240)]} 0 0
[junit4:junit4] 2> 179718 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10054 (-1437949371476869120)]} 0 1
[junit4:junit4] 2> 179720 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[54 (-1437949371477917696)]} 0 1
[junit4:junit4] 2> 179723 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10059 (1437949371481063424)]} 0 2
[junit4:junit4] 2> 179724 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[54 (-1437949371482112000)]} 0 1
[junit4:junit4] 2> 179728 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10059 (1437949371486306304)]} 0 2
[junit4:junit4] 2> 179729 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[59 (1437949371487354880)]} 0 2
[junit4:junit4] 2> 179734 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[59 (1437949371492597760)]} 0 2
[junit4:junit4] 2> 179751 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10055 (-1437949371511472128)]} 0 1
[junit4:junit4] 2> 179755 T273 C111 P30946 oass.SolrIndexSearcher.<init> Opening Searcher@2b6eec1 realtime
[junit4:junit4] 2> 179755 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10055 (-1437949371515666432)]} 0 0
[junit4:junit4] 2> 179756 T273 C111 P30946 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 179756 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[55 (-1437949371515666432)]} 0 1
[junit4:junit4] 2> 179756 T273 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 208
[junit4:junit4] 2> 179758 T298 C109 P30949 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 179758 T298 C109 P30949 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 179760 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[55 (-1437949371519860736)]} 0 1
[junit4:junit4] 2> 179760 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10060 (1437949371519860736)]} 0 2
[junit4:junit4] 2> 179761 T274 C111 P30946 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 179762 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 179763 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 179763 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 179763 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 179765 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[60 (1437949371525103616)]} 0 2
[junit4:junit4] 2> 179766 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10060 (1437949371526152192)]} 0 2
[junit4:junit4] 2> 179766 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 179767 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 179768 T298 C109 P30949 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920
[junit4:junit4] 2> 179769 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@44c0fc36 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f9777af) fullCopy=false
[junit4:junit4] 2> 179770 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[60 (1437949371530346496)]} 0 2
[junit4:junit4] 2> 179771 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179776 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179781 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179785 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 179788 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 179789 T260 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30949_rhm%2Fw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30949/rhm/w"}
[junit4:junit4] 2> 179790 T274 C111 P30946 oasc.SolrCore.execute [collection1] webapp=/rhm/w path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 179791 T298 C109 P30949 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 179793 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 179793 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 179794 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 179800 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10056 (-1437949371562852352)]} 0 0
[junit4:junit4] 2> 179802 T295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179802 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179802 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179802 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179803 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[56 (-1437949371565998080)]} 0 1
[junit4:junit4] 2> 179805 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10056 (-1437949371568095232)]} 0 1
[junit4:junit4] 2> 179808 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[56 (-1437949371571240960)]} 0 1
[junit4:junit4] 2> 179810 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10061 (1437949371572289536)]} 0 2
[junit4:junit4] 2> 179813 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[61 (1437949371575435264)]} 0 2
[junit4:junit4] 2> 179815 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10061 (1437949371577532416)]} 0 2
[junit4:junit4] 2> 179818 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[61 (1437949371580678144)]} 0 2
[junit4:junit4] 2> 179818 T298 C109 P30949 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 179819 T298 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 179823 T298 C109 P30949 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 179823 T298 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 179824 T298 C109 P30949 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 179827 T298 C109 P30949 oass.SolrIndexSearcher.<init> Opening Searcher@7142666b main
[junit4:junit4] 2> 179828 T297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7142666b main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c38/14)}
[junit4:junit4] 2> 179829 T298 C109 P30949 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920;done=true>>]
[junit4:junit4] 2> 179829 T298 C109 P30949 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920
[junit4:junit4] 2> 179830 T298 C109 P30949 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index.20130615142943920
[junit4:junit4] 2> 179838 T298 C109 P30949 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 179838 T306 C109 P30949 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=5468
[junit4:junit4] 2> 179845 T306 C109 P30949 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 179847 T306 C109 P30949 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@a306a17 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3840fe2e; maxCacheMB=0.87890625 maxMergeSizeMB=0.919921875)),segFN=segments_3,generation=3}
[junit4:junit4] 2> 179847 T306 C109 P30949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 179848 T306 C109 P30949 oass.SolrIndexSearcher.<init> Opening Searcher@33f376d2 main
[junit4:junit4] 2> 179848 T306 C109 P30949 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 179849 T297 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33f376d2 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c38/14)}
[junit4:junit4] 2> 179850 T306 C109 P30949 oasup.LogUpdateProcessor.finish [collection1] {} 0 12
[junit4:junit4] 2> 179850 T306 C109 P30949 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=5468}
[junit4:junit4] 2> 179851 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 179851 T298 C109 P30949 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 179851 T298 C109 P30949 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 179854 T298 C109 P30949 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 179855 T237 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@a41ba9d
[junit4:junit4] 2> 179864 T237 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=38,cumulative_deletesById=16,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 179865 T237 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 179866 T237 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 179866 T237 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 179869 T237 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 179870 T237 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 179870 T237 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2;done=false>>]
[junit4:junit4] 2> 179871 T237 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2
[junit4:junit4] 2> 179871 T237 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index;done=false>>]
[junit4:junit4] 2> 179872 T237 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371335366311/jetty2/index
[junit4:junit4] 2> 179872 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10057 (-1437949371638349824)]} 0 0
[junit4:junit4] 2> 179874 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 179875 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[57 (-1437949371640446976)]} 0 1
[junit4:junit4] 2> 179876 T259 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 179876 T279 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 179875 T237 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/rhm/w,null}
[junit4:junit4] 2> 179877 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10057 (-1437949371643592704)]} 0 1
[junit4:junit4] 2> 179876 T279 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 179876 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 179877 T266 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 179880 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[57 (-1437949371646738432)]} 0 1
[junit4:junit4] 2> 179882 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10062 (1437949371647787008)]} 0 2
[junit4:junit4] 2> 179885 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[62 (1437949371650932736)]} 0 2
[junit4:junit4] 2> 179887 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10062 (1437949371653029888)]} 0 2
[junit4:junit4] 2> 179890 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[62 (1437949371656175616)]} 0 2
[junit4:junit4] 2> 179909 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10063 (1437949371676098560)]} 0 2
[junit4:junit4] 2> 179911 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[63 (1437949371678195712)]} 0 2
[junit4:junit4] 2> 179915 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10063 (1437949371682390016)]} 0 2
[junit4:junit4] 2> 179916 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[63 (1437949371683438592)]} 0 2
[junit4:junit4] 2> 179948 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10058 (-1437949371718041600)]} 0 1
[junit4:junit4] 2> 179950 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[58 (-1437949371719090176)]} 0 1
[junit4:junit4] 2> 179952 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10058 (-1437949371721187328)]} 0 1
[junit4:junit4] 2> 179954 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[58 (-1437949371724333056)]} 0 1
[junit4:junit4] 2> 179957 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10064 (1437949371726430208)]} 0 2
[junit4:junit4] 2> 179962 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[64 (1437949371728527360)]} 0 5
[junit4:junit4] 2> 179962 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10064 (1437949371730624512)]} 0 2
[junit4:junit4] 2> 179967 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[64 (1437949371736915968)]} 0 2
[junit4:junit4] 2> 180026 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10065 (1437949371798781952)]} 0 1
[junit4:junit4] 2> 180030 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[65 (1437949371802976256)]} 0 2
[junit4:junit4] 2> 180032 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10065 (1437949371805073408)]} 0 2
[junit4:junit4] 2> 180035 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[65 (1437949371808219136)]} 0 2
[junit4:junit4] 2> 180099 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10066 (1437949371875328000)]} 0 1
[junit4:junit4] 2> 180101 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[66 (1437949371877425152)]} 0 2
[junit4:junit4] 2> 180105 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10066 (1437949371881619456)]} 0 2
[junit4:junit4] 2> 180106 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[66 (1437949371882668032)]} 0 2
[junit4:junit4] 2> 180132 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10067 (1437949371909931008)]} 0 2
[junit4:junit4] 2> 180133 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[67 (1437949371910979584)]} 0 2
[junit4:junit4] 2> 180137 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10067 (1437949371915173888)]} 0 2
[junit4:junit4] 2> 180138 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[67 (1437949371916222464)]} 0 2
[junit4:junit4] 2> 180217 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10059 (-1437949372000108544)]} 0 1
[junit4:junit4] 2> 180219 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[59 (-1437949372001157120)]} 0 1
[junit4:junit4] 2> 180221 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10059 (-1437949372003254272)]} 0 1
[junit4:junit4] 2> 180223 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[59 (-1437949372006400000)]} 0 1
[junit4:junit4] 2> 180226 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10068 (1437949372008497152)]} 0 2
[junit4:junit4] 2> 180228 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[68 (1437949372010594304)]} 0 2
[junit4:junit4] 2> 180231 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10068 (1437949372013740032)]} 0 2
[junit4:junit4] 2> 180233 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[68 (1437949372015837184)]} 0 2
[junit4:junit4] 2> 180301 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10069 (1437949372087140352)]} 0 1
[junit4:junit4] 2> 180303 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[69 (1437949372089237504)]} 0 2
[junit4:junit4] 2> 180306 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10069 (1437949372092383232)]} 0 2
[junit4:junit4] 2> 180308 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[69 (1437949372094480384)]} 0 2
[junit4:junit4] 2> 180310 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10060 (-1437949372096577536)]} 0 1
[junit4:junit4] 2> 180312 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[60 (-1437949372098674688)]} 0 1
[junit4:junit4] 2> 180314 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10060 (-1437949372101820416)]} 0 1
[junit4:junit4] 2> 180316 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[60 (-1437949372102868992)]} 0 1
[junit4:junit4] 2> 180319 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10070 (1437949372104966144)]} 0 2
[junit4:junit4] 2> 180321 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[70 (1437949372108111872)]} 0 2
[junit4:junit4] 2> 180324 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10070 (1437949372111257600)]} 0 2
[junit4:junit4] 2> 180326 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[70 (1437949372113354752)]} 0 2
[junit4:junit4] 2> 180337 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10062 (-1437949372124889088)]} 0 1
[junit4:junit4] 2> 180339 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[62 (-1437949372126986240)]} 0 0
[junit4:junit4] 2> 180341 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10062 (-1437949372130131968)]} 0 1
[junit4:junit4] 2> 180342 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[62 (-1437949372130131969)]} 0 1
[junit4:junit4] 2> 180346 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10071 (1437949372134326272)]} 0 2
[junit4:junit4] 2> 180347 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[71 (1437949372135374848)]} 0 2
[junit4:junit4] 2> 180351 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10071 (1437949372139569152)]} 0 2
[junit4:junit4] 2> 180352 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[71 (1437949372140617728)]} 0 2
[junit4:junit4] 2> 180424 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10072 (1437949372216115200)]} 0 2
[junit4:junit4] 2> 180424 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[72 (1437949372216115201)]} 0 1
[junit4:junit4] 2> 180429 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10072 (1437949372221358080)]} 0 2
[junit4:junit4] 2> 180430 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[72 (1437949372222406656)]} 0 2
[junit4:junit4] 2> 180483 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10073 (1437949372277981184)]} 0 2
[junit4:junit4] 2> 180484 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[73 (1437949372279029760)]} 0 2
[junit4:junit4] 2> 180488 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10073 (1437949372283224064)]} 0 2
[junit4:junit4] 2> 180489 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[73 (1437949372284272640)]} 0 2
[junit4:junit4] 2> 180550 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10063 (-1437949372349284352)]} 0 1
[junit4:junit4] 2> 180551 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[63 (-1437949372350332928)]} 0 1
[junit4:junit4] 2> 180553 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10063 (-1437949372352430080)]} 0 0
[junit4:junit4] 2> 180554 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[63 (-1437949372353478656)]} 0 0
[junit4:junit4] 2> 180557 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10074 (1437949372355575808)]} 0 1
[junit4:junit4] 2> 180558 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[74 (1437949372356624384)]} 0 1
[junit4:junit4] 2> 180561 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10074 (1437949372359770112)]} 0 1
[junit4:junit4] 2> 180562 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[74 (1437949372360818688)]} 0 1
[junit4:junit4] 2> 180582 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10075 (1437949372381790208)]} 0 1
[junit4:junit4] 2> 180584 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[75 (1437949372382838784)]} 0 2
[junit4:junit4] 2> 180586 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10075 (1437949372385984512)]} 0 1
[junit4:junit4] 2> 180589 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[75 (1437949372389130240)]} 0 2
[junit4:junit4] 2> 180608 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10064 (-1437949372409053184)]} 0 1
[junit4:junit4] 2> 180610 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[64 (-1437949372411150336)]} 0 1
[junit4:junit4] 2> 180612 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[10064 (-1437949372413247488)]} 0 1
[junit4:junit4] 2> 180613 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {delete=[64 (-1437949372415344640)]} 0 0
[junit4:junit4] 2> 180616 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10076 (1437949372416393216)]} 0 2
[junit4:junit4] 2> 180617 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[76 (1437949372418490368)]} 0 1
[junit4:junit4] 2> 180619 T271 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10076 (1437949372420587520)]} 0 1
[junit4:junit4] 2> 180621 T270 C111 P30946 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[76 (1437949372422684672)]} 0 1
[junit4:junit4] 2> 180703 T249 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[10077 (1437949372508667904)]} 0 2
[junit4:junit4] 2> 180705 T250 C110 P30941 oasup.LogUpdateProcessor.finish [collection1] webapp=/rhm/w path=/update params={wt=javabin&version=2} {add=[77 (1437949372510765
[...truncated too long message...]
loseSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1839970 T3763 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1839970 T3763 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4/index;done=false>>]
[junit4:junit4] 2> 1839971 T3763 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4/index
[junit4:junit4] 2> 1839971 T3763 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4;done=false>>]
[junit4:junit4] 2> 1839971 T3763 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371336820693/jetty4
[junit4:junit4] 2> 1839972 T3763 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 1839972 T3763 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election/89871929881460748-5-n_0000000001
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:260)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1839973 T3763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1839973 T3763 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1839973 T3763 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1839974 T3763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1839974 T3763 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1839974 T3763 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=uk, timezone=Pacific/Yap
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=255022080,total=491782144
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestOmitPositions, TestDefaultSimilarityFactory, IndexReaderFactoryTest, BasicDistributedZk2Test, ClusterStateTest, HighlighterTest, TestSystemIdResolver, TestStressLucene, TestZkChroot, TestJmxIntegration, RequiredFieldsTest, TestIndexingPerformance, SortByFunctionTest, DistanceFunctionTest, UniqFieldsUpdateProcessorFactoryTest, TestCopyFieldCollectionResource, FieldAnalysisRequestHandlerTest, BasicZkTest, TestCSVResponseWriter, TestDistributedGrouping, TestBinaryField, TestHashPartitioner, DefaultValueUpdateProcessorTest, TestUpdate, TestArbitraryIndexDir, TestSchemaNameResource, TestRandomFaceting, TestReload, TestRecovery, SampleTest, TestSuggestSpellingConverter, DirectSolrSpellCheckerTest, TestFieldResource, TestManagedSchemaFieldResource, TestComponentsName, SignatureUpdateProcessorFactoryTest, TestSolrQueryParserResource, JSONWriterTest, TestCoreDiscovery, TestPostingsSolrHighlighter, IndexSchemaRuntimeFieldTest, TestLuceneMatchVersion, TestClassNameShortening, TestAtomicUpdateErrorCases, TestFastOutputStream, MultiTermTest, TestUniqueKeyFieldResource, ShardRoutingCustomTest, FileUtilsTest, SolrCmdDistributorTest, TestSolr4Spatial, TestShardHandlerFactory, PingRequestHandlerTest, SolrPluginUtilsTest, StandardRequestHandlerTest, TestCSVLoader, TestDefaultSearchFieldResource, TestQueryTypes, TestNumberUtils, TestDynamicFieldCollectionResource, SpellCheckCollatorTest, TestSolrXMLSerializer, TestPartialUpdateDeduplication, TestLMJelinekMercerSimilarityFactory, CurrencyFieldOpenExchangeTest, TestFiltering, NotRequiredUniqueKeyTest, ChaosMonkeySafeLeaderTest, TestIndexSearcher, TestRTGBase, SuggesterWFSTTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, ZkCLITest, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestStressReorder, HardAutoCommitTest, TestRangeQuery, SolrCoreTest, StatsComponentTest, TestGroupingSearch, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SolrRequestParserTest, DocValuesTest, SuggesterTest, NoCacheHeaderTest, TestPseudoReturnFields, TestRemoteStreaming, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, XmlUpdateRequestHandlerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, TestCollationField, QueryParsingTest, JsonLoaderTest, CSVRequestHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, TestPropInject, UpdateParamsTest, TestSolrIndexConfig, CopyFieldTest, SolrIndexConfigTest, NumericFieldsTest, TestElisionMultitermQuery, TestPhraseSuggestions, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, DOMUtilTest, TestSolrJ, TestUtils, SliceStateTest, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed on J1 in 223.70s, 1 test, 1 error <<< FAILURES!
[...truncated 54 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:386: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:366: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:887: There were test failures: 296 suites, 1243 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 52 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4068 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4068/
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:20986 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:20986 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([7262B9B042D2C205:F38437A8358DA239]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 482 from http://127.0.0.1:41104/collection1lastClient and got 476 from http://127.0.0.1:33657/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 482 from http://127.0.0.1:41104/collection1lastClient and got 476 from http://127.0.0.1:33657/collection1
at __randomizedtesting.SeedInfo.seed([7262B9B042D2C205:F38437A8358DA239]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9542 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 946420 T1801 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 946428 T1801 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyShardSplitTest-1371299031648
[junit4:junit4] 2> 946430 T1801 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 946430 T1802 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 946531 T1801 oasc.ZkTestServer.run start zk server on port:34237
[junit4:junit4] 2> 946532 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 946536 T1808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d61e46a name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 946536 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 946537 T1801 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 946555 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 946557 T1810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14547f9a name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 946557 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 946558 T1801 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 946562 T1801 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 946573 T1801 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 946575 T1801 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 946579 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 946579 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 946592 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 946592 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 946707 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 946708 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 946719 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 946720 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 946723 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 946724 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 946727 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 946728 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 946731 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 946731 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 946734 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 946735 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 946738 T1801 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 946739 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 947020 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 947026 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11430
[junit4:junit4] 2> 947026 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 947027 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 947028 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968
[junit4:junit4] 2> 947028 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/solr.xml
[junit4:junit4] 2> 947029 T1801 oasc.CoreContainer.<init> New CoreContainer 1424141568
[junit4:junit4] 2> 947029 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/'
[junit4:junit4] 2> 947030 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/'
[junit4:junit4] 2> 947160 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 947161 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 947162 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 947162 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 947163 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 947163 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 947164 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 947164 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 947165 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 947166 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 947183 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 947184 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 947184 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 947185 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 947213 T1821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43576488 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 947214 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 947220 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 947233 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 947280 T1823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3069cf63 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 947280 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 947295 T1801 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 947394 T1801 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 947413 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 947415 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11430_
[junit4:junit4] 2> 947461 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11430_
[junit4:junit4] 2> 947465 T1801 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 947480 T1801 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 947483 T1801 oasc.Overseer.start Overseer (id=89869453338607619-127.0.0.1:11430_-n_0000000000) starting
[junit4:junit4] 2> 947494 T1801 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 947505 T1825 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 947506 T1801 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 947509 T1801 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 947511 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 947513 T1824 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 947517 T1826 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 947517 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 949017 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 949018 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:11430_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:11430"}
[junit4:junit4] 2> 949018 T1824 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 949018 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 949031 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 949521 T1826 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1
[junit4:junit4] 2> 949521 T1826 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 949522 T1826 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 949522 T1826 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 949524 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/'
[junit4:junit4] 2> 949525 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/lib/README' to classloader
[junit4:junit4] 2> 949526 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 949576 T1826 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 949634 T1826 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 949735 T1826 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 949742 T1826 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 950331 T1826 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 950336 T1826 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 950338 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 950343 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 950368 T1826 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 950368 T1826 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1371299031968/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/
[junit4:junit4] 2> 950369 T1826 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 950369 T1826 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 950370 T1826 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data
[junit4:junit4] 2> 950370 T1826 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/index/
[junit4:junit4] 2> 950371 T1826 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 950371 T1826 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/index
[junit4:junit4] 2> 950376 T1826 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_1,generation=1}
[junit4:junit4] 2> 950376 T1826 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 950379 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 950380 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 950381 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 950381 T1826 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 950382 T1826 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 950382 T1826 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 950382 T1826 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 950383 T1826 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 950383 T1826 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 950400 T1826 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 950407 T1826 oass.SolrIndexSearcher.<init> Opening Searcher@710cac4c main
[junit4:junit4] 2> 950408 T1826 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/control/data/tlog
[junit4:junit4] 2> 950409 T1826 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 950409 T1826 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 950413 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@710cac4c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 950415 T1826 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 950415 T1826 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11430 collection:control_collection shard:shard1
[junit4:junit4] 2> 950416 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 950448 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 950460 T1826 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 950461 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 950461 T1826 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11430/collection1/
[junit4:junit4] 2> 950461 T1826 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 950462 T1826 oasc.SyncStrategy.syncToMe http://127.0.0.1:11430/collection1/ has no replicas
[junit4:junit4] 2> 950462 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11430/collection1/
[junit4:junit4] 2> 950462 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 950536 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 950553 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.register We are http://127.0.0.1:11430/collection1/ and leader is http://127.0.0.1:11430/collection1/
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11430
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 950576 T1826 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 950577 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 950578 T1826 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 950579 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 950580 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 950581 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 950597 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 950598 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 950600 T1830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@249843d6 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 950601 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 950603 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 950605 T1801 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 950873 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 950876 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:20986
[junit4:junit4] 2> 950877 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 950877 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 950878 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825
[junit4:junit4] 2> 950878 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/solr.xml
[junit4:junit4] 2> 950879 T1801 oasc.CoreContainer.<init> New CoreContainer 1196405988
[junit4:junit4] 2> 950880 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/'
[junit4:junit4] 2> 950880 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/'
[junit4:junit4] 2> 951011 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 951012 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 951012 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 951013 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 951013 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 951014 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 951014 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 951015 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 951015 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 951016 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 951032 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 951032 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 951033 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 951034 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 951037 T1841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f801b05 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 951038 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 951053 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 951064 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 951066 T1843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26df2e name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 951066 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 951072 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 952067 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 952068 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:11430_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:11430"}
[junit4:junit4] 2> 952105 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20986_
[junit4:junit4] 2> 952180 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20986_
[junit4:junit4] 2> 952206 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952206 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952206 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952262 T1843 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 952262 T1830 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 952262 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 952269 T1823 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 952273 T1844 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 952273 T1844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 953764 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 953765 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20986_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20986"}
[junit4:junit4] 2> 953765 T1824 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 953765 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 953778 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 953778 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 953778 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 954277 T1844 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1
[junit4:junit4] 2> 954277 T1844 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 954278 T1844 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 954278 T1844 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 954280 T1844 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/'
[junit4:junit4] 2> 954281 T1844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/lib/README' to classloader
[junit4:junit4] 2> 954282 T1844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 954335 T1844 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 954397 T1844 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 954498 T1844 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 954505 T1844 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 955086 T1844 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 955090 T1844 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 955092 T1844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 955097 T1844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 955121 T1844 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 955122 T1844 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1371299035825/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/
[junit4:junit4] 2> 955122 T1844 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 955122 T1844 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 955123 T1844 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1
[junit4:junit4] 2> 955123 T1844 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/index/
[junit4:junit4] 2> 955124 T1844 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 955125 T1844 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/index
[junit4:junit4] 2> 955129 T1844 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 955129 T1844 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 955132 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 955133 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 955134 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 955134 T1844 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 955135 T1844 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 955135 T1844 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 955136 T1844 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 955136 T1844 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 955137 T1844 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 955153 T1844 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 955161 T1844 oass.SolrIndexSearcher.<init> Opening Searcher@4c6923b5 main
[junit4:junit4] 2> 955161 T1844 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty1/tlog
[junit4:junit4] 2> 955162 T1844 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 955163 T1844 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 955167 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c6923b5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 955170 T1844 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 955170 T1844 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:20986 collection:collection1 shard:shard1
[junit4:junit4] 2> 955171 T1844 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 955187 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 955190 T1844 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 955190 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 955190 T1844 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 955190 T1844 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 955191 T1844 oasc.SyncStrategy.syncToMe http://127.0.0.1:20986/collection1/ has no replicas
[junit4:junit4] 2> 955191 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 955191 T1844 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 955282 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 955299 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 955299 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 955299 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.register We are http://127.0.0.1:20986/collection1/ and leader is http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:20986
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 955350 T1844 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 955351 T1844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 955352 T1844 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 955353 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 955354 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 955355 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 955612 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 955615 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:62105
[junit4:junit4] 2> 955615 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 955616 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 955616 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590
[junit4:junit4] 2> 955617 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/solr.xml
[junit4:junit4] 2> 955617 T1801 oasc.CoreContainer.<init> New CoreContainer 1506313995
[junit4:junit4] 2> 955618 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/'
[junit4:junit4] 2> 955618 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/'
[junit4:junit4] 2> 955739 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 955740 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 955740 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 955741 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 955741 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 955742 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 955742 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 955743 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 955743 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 955744 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 955760 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 955761 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 955761 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 955762 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 955766 T1857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bcf40e0 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 955767 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 955769 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 955782 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 955784 T1859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50797dd7 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 955784 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 955796 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 956800 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62105_
[junit4:junit4] 2> 956802 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:62105_
[junit4:junit4] 2> 956806 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 956806 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 956806 T1830 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956806 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 956807 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956807 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:20986_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:20986"}
[junit4:junit4] 2> 956806 T1859 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956817 T1843 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956820 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956820 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956820 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 956823 T1860 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 956823 T1860 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 956832 T1823 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 956833 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958333 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 958334 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:62105_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:62105"}
[junit4:junit4] 2> 958335 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 958335 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 958348 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958348 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958348 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958348 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 958833 T1860 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1
[junit4:junit4] 2> 958833 T1860 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 958834 T1860 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 958834 T1860 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 958836 T1860 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/'
[junit4:junit4] 2> 958837 T1860 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/lib/README' to classloader
[junit4:junit4] 2> 958838 T1860 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 958892 T1860 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 958955 T1860 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 959056 T1860 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 959063 T1860 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 959644 T1860 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 959648 T1860 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 959651 T1860 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 959656 T1860 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 959680 T1860 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 959680 T1860 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1371299040590/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/
[junit4:junit4] 2> 959681 T1860 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 959681 T1860 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 959682 T1860 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2
[junit4:junit4] 2> 959682 T1860 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/index/
[junit4:junit4] 2> 959683 T1860 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 959683 T1860 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/index
[junit4:junit4] 2> 959687 T1860 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_1,generation=1}
[junit4:junit4] 2> 959687 T1860 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 959690 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 959691 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 959692 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 959692 T1860 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 959693 T1860 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 959693 T1860 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 959694 T1860 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 959694 T1860 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 959695 T1860 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 959710 T1860 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 959717 T1860 oass.SolrIndexSearcher.<init> Opening Searcher@3f73983a main
[junit4:junit4] 2> 959717 T1860 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty2/tlog
[junit4:junit4] 2> 959718 T1860 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 959718 T1860 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 959723 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f73983a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 959725 T1860 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 959726 T1860 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:62105 collection:collection1 shard:shard2
[junit4:junit4] 2> 959726 T1860 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 959744 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 959747 T1860 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 959747 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 959747 T1860 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 959747 T1860 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 959748 T1860 oasc.SyncStrategy.syncToMe http://127.0.0.1:62105/collection1/ has no replicas
[junit4:junit4] 2> 959748 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 959748 T1860 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 959852 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 959868 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959868 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959868 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959868 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.register We are http://127.0.0.1:62105/collection1/ and leader is http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:62105
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 959906 T1860 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 959907 T1860 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 959909 T1860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 959910 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 959910 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 959911 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 960173 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 960177 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:41605
[junit4:junit4] 2> 960178 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 960178 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 960179 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144
[junit4:junit4] 2> 960179 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/solr.xml
[junit4:junit4] 2> 960180 T1801 oasc.CoreContainer.<init> New CoreContainer 578983594
[junit4:junit4] 2> 960180 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/'
[junit4:junit4] 2> 960181 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/'
[junit4:junit4] 2> 960302 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 960302 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 960303 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 960303 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 960304 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 960304 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 960305 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 960305 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 960306 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 960306 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 960320 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 960321 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 960322 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 960323 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 960326 T1873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a67455b name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 960326 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 960342 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 960353 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 960355 T1875 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d4cc7fa name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 960355 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 960360 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 961364 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41605_
[junit4:junit4] 2> 961366 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41605_
[junit4:junit4] 2> 961370 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 961370 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 961370 T1859 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961370 T1875 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961370 T1830 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961371 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961371 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961372 T1823 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961379 T1843 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 961380 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 961381 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:62105_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:62105"}
[junit4:junit4] 2> 961384 T1876 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 961385 T1876 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 961394 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 961394 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962900 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 962901 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41605_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41605"}
[junit4:junit4] 2> 962901 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 962901 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 962925 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 962925 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 963396 T1876 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1
[junit4:junit4] 2> 963396 T1876 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 963397 T1876 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 963397 T1876 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 963399 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/'
[junit4:junit4] 2> 963400 T1876 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/lib/README' to classloader
[junit4:junit4] 2> 963401 T1876 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 963454 T1876 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 963516 T1876 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 963617 T1876 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 963624 T1876 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 964184 T1876 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 964187 T1876 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 964189 T1876 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 964193 T1876 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 964214 T1876 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 964214 T1876 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1371299045144/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/
[junit4:junit4] 2> 964214 T1876 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 964215 T1876 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 964215 T1876 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3
[junit4:junit4] 2> 964216 T1876 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index/
[junit4:junit4] 2> 964216 T1876 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 964217 T1876 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index
[junit4:junit4] 2> 964220 T1876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 964220 T1876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 964222 T1876 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 964222 T1876 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 964223 T1876 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 964223 T1876 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 964224 T1876 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 964224 T1876 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 964224 T1876 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 964225 T1876 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 964225 T1876 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 964236 T1876 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 964242 T1876 oass.SolrIndexSearcher.<init> Opening Searcher@349da0e2 main
[junit4:junit4] 2> 964242 T1876 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/tlog
[junit4:junit4] 2> 964243 T1876 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 964243 T1876 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 964246 T1877 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@349da0e2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 964248 T1876 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 964248 T1876 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41605 collection:collection1 shard:shard1
[junit4:junit4] 2> 964252 T1876 oasc.ZkController.register We are http://127.0.0.1:41605/collection1/ and leader is http://127.0.0.1:20986/collection1/
[junit4:junit4] 2> 964252 T1876 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41605
[junit4:junit4] 2> 964252 T1876 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 964252 T1876 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C712 name=collection1 org.apache.solr.core.SolrCore@65177b81 url=http://127.0.0.1:41605/collection1 node=127.0.0.1:41605_ C712_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41605_, base_url=http://127.0.0.1:41605}
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 964253 T1876 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 964253 T1878 C712 P41605 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 964254 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 964254 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 964255 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 964255 T1878 C712 P41605 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 964272 T1833 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 964433 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 964434 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41605_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41605"}
[junit4:junit4] 2> 964447 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964447 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 964465 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 964468 T1801 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:10317
[junit4:junit4] 2> 964468 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 964469 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 964469 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489
[junit4:junit4] 2> 964469 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/solr.xml
[junit4:junit4] 2> 964470 T1801 oasc.CoreContainer.<init> New CoreContainer 1052877441
[junit4:junit4] 2> 964470 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/'
[junit4:junit4] 2> 964470 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/'
[junit4:junit4] 2> 964574 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 964575 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 964575 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 964575 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 964576 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 964576 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 964576 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 964577 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 964577 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 964577 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 964589 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 964589 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34237/solr
[junit4:junit4] 2> 964589 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 964590 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964593 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a804510 name:ZooKeeperConnection Watcher:127.0.0.1:34237 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964594 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964596 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 964609 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964610 T1892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5db02d31 name:ZooKeeperConnection Watcher:127.0.0.1:34237/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964611 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964623 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 965273 T1833 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 965273 T1833 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:41605_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 965626 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:10317_
[junit4:junit4] 2> 965628 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:10317_
[junit4:junit4] 2> 965632 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 965632 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 965632 T1859 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965632 T1892 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965632 T1830 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965632 T1875 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965633 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965634 T1843 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965633 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965634 T1823 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 965633 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965648 T1893 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 965648 T1893 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 965956 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 965957 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10317_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10317"}
[junit4:junit4] 2> 965957 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 965958 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 965973 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 965973 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 966650 T1893 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1
[junit4:junit4] 2> 966650 T1893 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 966651 T1893 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 966651 T1893 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 966653 T1893 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/'
[junit4:junit4] 2> 966655 T1893 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/lib/README' to classloader
[junit4:junit4] 2> 966655 T1893 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 966710 T1893 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 966771 T1893 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 966872 T1893 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 966881 T1893 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C713 name=collection1 org.apache.solr.core.SolrCore@65177b81 url=http://127.0.0.1:41605/collection1 node=127.0.0.1:41605_ C713_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41605_, base_url=http://127.0.0.1:41605}
[junit4:junit4] 2> 967275 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:20986/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 967275 T1878 C713 P41605 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41605 START replicas=[http://127.0.0.1:20986/collection1/] nUpdates=100
[junit4:junit4] 2> 967276 T1878 C713 P41605 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 967277 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 967277 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 967277 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 967278 T1878 C713 P41605 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 967278 T1878 C713 P41605 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:20986/collection1/. core=collection1
[junit4:junit4] 2> 967278 T1878 C713 P41605 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C714 name=collection1 org.apache.solr.core.SolrCore@790af722 url=http://127.0.0.1:20986/collection1 node=127.0.0.1:20986_ C714_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:20986_, base_url=http://127.0.0.1:20986, leader=true}
[junit4:junit4] 2> 967286 T1834 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 967291 T1835 C714 P20986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 967294 T1835 C714 P20986 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 967295 T1835 C714 P20986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 967296 T1835 C714 P20986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_2,generation=2}
[junit4:junit4] 2> 967296 T1835 C714 P20986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 967297 T1835 C714 P20986 oass.SolrIndexSearcher.<init> Opening Searcher@7154583f realtime
[junit4:junit4] 2> 967297 T1835 C714 P20986 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 967298 T1835 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 967299 T1878 C713 P41605 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 967299 T1878 C713 P41605 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 967302 T1836 C714 P20986 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 967302 T1836 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 967303 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 967303 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 967303 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 967306 T1836 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 967307 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 967308 T1878 C713 P41605 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526
[junit4:junit4] 2> 967308 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d118d8c lockFactory=org.apache.lucene.store.NativeFSLockFactory@3405ec2e) fullCopy=false
[junit4:junit4] 2> 967311 T1836 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 967312 T1878 C713 P41605 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 967313 T1878 C713 P41605 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 967313 T1878 C713 P41605 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 967316 T1878 C713 P41605 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 967316 T1878 C713 P41605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 967317 T1878 C713 P41605 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 967317 T1878 C713 P41605 oass.SolrIndexSearcher.<init> Opening Searcher@79c6e455 main
[junit4:junit4] 2> 967318 T1877 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c6e455 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 967319 T1878 C713 P41605 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526;done=true>>]
[junit4:junit4] 2> 967319 T1878 C713 P41605 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526
[junit4:junit4] 2> 967319 T1878 C713 P41605 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty3/index.20130616012412526
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 967320 T1878 C713 P41605 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 967328 T1878 C713 P41605 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 967445 T1893 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 967449 T1893 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 967452 T1893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 967457 T1893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 967480 T1893 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 967480 T1893 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1371299049489/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/
[junit4:junit4] 2> 967481 T1893 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56e746d4
[junit4:junit4] 2> 967481 T1893 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 967482 T1893 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4
[junit4:junit4] 2> 967482 T1893 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index/
[junit4:junit4] 2> 967483 T1893 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 967483 T1893 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index
[junit4:junit4] 2> 967487 T1893 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_1,generation=1}
[junit4:junit4] 2> 967488 T1893 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 967491 T1893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 967491 T1893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 967492 T1893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 967493 T1893 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 967494 T1893 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 967494 T1893 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 967495 T1893 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 967495 T1893 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 967496 T1893 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 967497 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967498 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41605_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41605"}
[junit4:junit4] 2> 967512 T1893 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 967519 T1893 oass.SolrIndexSearcher.<init> Opening Searcher@c13d219 main
[junit4:junit4] 2> 967519 T1893 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/tlog
[junit4:junit4] 2> 967521 T1893 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 967521 T1893 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 967526 T1896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c13d219 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 967528 T1893 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 967528 T1893 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:10317 collection:collection1 shard:shard2
[junit4:junit4] 2> 967539 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967539 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 967548 T1893 oasc.ZkController.register We are http://127.0.0.1:10317/collection1/ and leader is http://127.0.0.1:62105/collection1/
[junit4:junit4] 2> 967548 T1893 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:10317
[junit4:junit4] 2> 967549 T1893 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 967549 T1893 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C715 name=collection1 org.apache.solr.core.SolrCore@4fc79219 url=http://127.0.0.1:10317/collection1 node=127.0.0.1:10317_ C715_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:10317_, base_url=http://127.0.0.1:10317}
[junit4:junit4] 2> 967550 T1897 C715 P10317 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 967550 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 967550 T1893 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967550 T1897 C715 P10317 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 967551 T1897 C715 P10317 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 967551 T1801 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 967552 T1897 C715 P10317 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 967552 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 967553 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 967577 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967578 T1849 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 967578 T1801 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 967579 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 967579 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 968581 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 969050 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 969051 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10317_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10317"}
[junit4:junit4] 2> 969056 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969056 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 969580 T1849 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 969580 T1849 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:10317_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 969583 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 970585 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C715_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:10317_, base_url=http://127.0.0.1:10317}
[junit4:junit4] 2> 971582 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:62105/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 971582 T1897 C715 P10317 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:10317 START replicas=[http://127.0.0.1:62105/collection1/] nUpdates=100
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 971583 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 971584 T1897 C715 P10317 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 971584 T1897 C715 P10317 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:62105/collection1/. core=collection1
[junit4:junit4] 2> 971585 T1897 C715 P10317 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 971587 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C716 name=collection1 org.apache.solr.core.SolrCore@723241fd url=http://127.0.0.1:62105/collection1 node=127.0.0.1:62105_ C716_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:62105_, base_url=http://127.0.0.1:62105, leader=true}
[junit4:junit4] 2> 971592 T1850 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 971598 T1851 C716 P62105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 971600 T1851 C716 P62105 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_1,generation=1}
[junit4:junit4] 2> 971601 T1851 C716 P62105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 971602 T1851 C716 P62105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_2,generation=2}
[junit4:junit4] 2> 971603 T1851 C716 P62105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 971603 T1851 C716 P62105 oass.SolrIndexSearcher.<init> Opening Searcher@1b9fa58d realtime
[junit4:junit4] 2> 971604 T1851 C716 P62105 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 971604 T1851 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 971605 T1897 C715 P10317 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 971605 T1897 C715 P10317 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 971607 T1852 C716 P62105 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 971607 T1852 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 971608 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 971608 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 971608 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 971611 T1852 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 971612 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 971613 T1897 C715 P10317 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831
[junit4:junit4] 2> 971613 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@5b0dcf11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fc7f569) fullCopy=false
[junit4:junit4] 2> 971616 T1852 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 971617 T1897 C715 P10317 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 971618 T1897 C715 P10317 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 971618 T1897 C715 P10317 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 971620 T1897 C715 P10317 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_2,generation=2}
[junit4:junit4] 2> 971620 T1897 C715 P10317 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 971620 T1897 C715 P10317 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 971621 T1897 C715 P10317 oass.SolrIndexSearcher.<init> Opening Searcher@2c294a8a main
[junit4:junit4] 2> 971622 T1896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c294a8a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831;done=true>>]
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1371299031647/jetty4/index.20130616012416831
[junit4:junit4] 2> 971623 T1897 C715 P10317 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 971624 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 971624 T1897 C715 P10317 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 971624 T1897 C715 P10317 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 971626 T1897 C715 P10317 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 972075 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 972076 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10317_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10317"}
[junit4:junit4] 2> 972115 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972115 T1875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 972589 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 972590 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C717 name=collection1 org.apache.solr.core.SolrCore@29366d6b url=http://127.0.0.1:11430/collection1 node=127.0.0.1:11430_ C717_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:11430_, base_url=http://127.0.0.1:11430, leader=true}
[junit4:junit4] 2> 972599 T1813 C717 P11430 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972602 T1813 C717 P11430 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_1,generation=1}
[junit4:junit4] 2> 972603 T1813 C717 P11430 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 972603 T1813 C717 P11430 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@593ab685 lockFactory=org.apache.lucene.store.NativeFSLockFactory@31e52610),segFN=segments_2,generation=2}
[junit4:junit4] 2> 972604 T1813 C717 P11430 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 972604 T1813 C717 P11430 oass.SolrIndexSearcher.<init> Opening Searcher@7d448d31 main
[junit4:junit4] 2> 972605 T1813 C717 P11430 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972606 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d448d31 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972607 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 972619 T1837 C714 P20986 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972621 T1837 C714 P20986 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5bb125c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2514291a),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972621 T1837 C714 P20986 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972622 T1837 C714 P20986 oass.SolrIndexSearcher.<init> Opening Searcher@2f780015 main
[junit4:junit4] 2> 972623 T1837 C714 P20986 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972623 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f780015 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972624 T1837 C714 P20986 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:41605/collection1/, StdNode: http://127.0.0.1:62105/collection1/, StdNode: http://127.0.0.1:10317/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 972627 T1853 C716 P62105 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972629 T1853 C716 P62105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@785253f6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6df6daad),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972630 T1853 C716 P62105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972630 T1853 C716 P62105 oass.SolrIndexSearcher.<init> Opening Searcher@14c640c main
[junit4:junit4] 2> 972631 T1853 C716 P62105 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972632 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14c640c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972632 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> C713_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41605_, base_url=http://127.0.0.1:41605}
[junit4:junit4] 2> 972635 T1865 C713 P41605 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C715_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:10317_, base_url=http://127.0.0.1:10317}
[junit4:junit4] 2> 972635 T1882 C715 P10317 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 972637 T1865 C713 P41605 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4698405d lockFactory=org.apache.lucene.store.NativeFSLockFactory@3a1f881b),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972637 T1882 C715 P10317 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@77aaf4c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee519ba),segFN=segments_3,generation=3}
[junit4:junit4] 2> 972637 T1865 C713 P41605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972638 T1882 C715 P10317 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 972639 T1865 C713 P41605 oass.SolrIndexSearcher.<init> Opening Searcher@774c3fd9 main
[junit4:junit4] 2> 972640 T1882 C715 P10317 oass.SolrIndexSearcher.<init> Opening Searcher@38cb4af0 main
[junit4:junit4] 2> 972640 T1865 C713 P41605 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972641 T1877 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@774c3fd9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972640 T1882 C715 P10317 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 972641 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 972641 T1896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38cb4af0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 972642 T1882 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 972643 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4] 2> 972644 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 972647 T1838 C714 P20986 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 972649 T1866 C713 P41605 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 972651 T1854 C716 P62105 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 972653 T1883 C715 P10317 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 974659 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437911283009585152)} 0 2
[junit4:junit4] 2> 974668 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1437911283013779456&update.from=http://127.0.0.1:62105/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437911283013779456)} 0 2
[junit4:junit4] 2> 974671 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1437911283017973760&update.from=http://127.0.0.1:20986/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437911283017973760)} 0 1
[junit4:junit4] 2> 974672 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1437911283017973760)} 0 6
[junit4:junit4] 2> 974673 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437911283013779456)} 0 12
[junit4:junit4] 2> 974677 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1437911283028459520)]} 0 1
[junit4:junit4] 2> 974684 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1437911283032653824)]} 0 1
[junit4:junit4] 2> 974684 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1437911283032653824)]} 0 4
[junit4:junit4] 2> 974687 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1437911283039993856)]} 0 0
[junit4:junit4] 2> 974695 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1437911283045236736)]} 0 0
[junit4:junit4] 2> 974696 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1437911283045236736)]} 0 4
[junit4:junit4] 2> 974697 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 7
[junit4:junit4] 2> 974699 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1437911283052576768)]} 0 0
[junit4:junit4] 2> 974706 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1437911283055722496)]} 0 1
[junit4:junit4] 2> 974706 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1437911283055722496)]} 0 4
[junit4:junit4] 2> 974709 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1437911283063062528)]} 0 0
[junit4:junit4] 2> 974715 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1437911283066208256)]} 0 0
[junit4:junit4] 2> 974716 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1437911283066208256)]} 0 4
[junit4:junit4] 2> 974719 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1437911283073548288)]} 0 0
[junit4:junit4] 2> 974727 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1437911283078791168)]} 0 0
[junit4:junit4] 2> 974728 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1437911283078791168)]} 0 4
[junit4:junit4] 2> 974729 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4]} 0 7
[junit4:junit4] 2> 974732 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1437911283086131200)]} 0 1
[junit4:junit4] 2> 974738 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1437911283089276928)]} 0 1
[junit4:junit4] 2> 974738 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1437911283089276928)]} 0 4
[junit4:junit4] 2> 974741 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1437911283096616960)]} 0 0
[junit4:junit4] 2> 974749 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1437911283101859840)]} 0 0
[junit4:junit4] 2> 974750 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1437911283101859840)]} 0 4
[junit4:junit4] 2> 974751 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6]} 0 7
[junit4:junit4] 2> 974754 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1437911283109199872)]} 0 1
[junit4:junit4] 2> 974761 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1437911283113394176)]} 0 1
[junit4:junit4] 2> 974761 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1437911283113394176)]} 0 4
[junit4:junit4] 2> 974765 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1437911283120734208)]} 0 1
[junit4:junit4] 2> 974771 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1437911283124928512)]} 0 1
[junit4:junit4] 2> 974771 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1437911283124928512)]} 0 4
[junit4:junit4] 2> 974774 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1437911283131219968)]} 0 0
[junit4:junit4] 2> 974781 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1437911283134365696)]} 0 1
[junit4:junit4] 2> 974781 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1437911283134365696)]} 0 4
[junit4:junit4] 2> 974785 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1437911283141705728)]} 0 1
[junit4:junit4] 2> 974791 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1437911283144851456)]} 0 1
[junit4:junit4] 2> 974791 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1437911283144851456)]} 0 4
[junit4:junit4] 2> 974794 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1437911283152191488)]} 0 0
[junit4:junit4] 2> 974800 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1437911283155337216)]} 0 0
[junit4:junit4] 2> 974801 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1437911283155337216)]} 0 4
[junit4:junit4] 2> 974804 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1437911283162677248)]} 0 0
[junit4:junit4] 2> 974812 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1437911283167920128)]} 0 0
[junit4:junit4] 2> 974813 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[12 (1437911283167920128)]} 0 4
[junit4:junit4] 2> 974814 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12]} 0 7
[junit4:junit4] 2> 974817 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1437911283176308736)]} 0 0
[junit4:junit4] 2> 974826 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1437911283181551616)]} 0 0
[junit4:junit4] 2> 974827 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1437911283181551616)]} 0 5
[junit4:junit4] 2> 974828 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13]} 0 8
[junit4:junit4] 2> 974831 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1437911283189940224)]} 0 1
[junit4:junit4] 2> 974838 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1437911283195183104)]} 0 0
[junit4:junit4] 2> 974839 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[14 (1437911283195183104)]} 0 4
[junit4:junit4] 2> 974840 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14]} 0 7
[junit4:junit4] 2> 974842 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1437911283202523136)]} 0 0
[junit4:junit4] 2> 974848 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1437911283205668864)]} 0 0
[junit4:junit4] 2> 974849 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1437911283205668864)]} 0 4
[junit4:junit4] 2> 974852 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1437911283213008896)]} 0 0
[junit4:junit4] 2> 974858 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1437911283216154624)]} 0 0
[junit4:junit4] 2> 974859 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1437911283216154624)]} 0 4
[junit4:junit4] 2> 974862 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1437911283223494656)]} 0 0
[junit4:junit4] 2> 974868 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1437911283226640384)]} 0 0
[junit4:junit4] 2> 974869 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1437911283226640384)]} 0 4
[junit4:junit4] 2> 974872 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1437911283233980416)]} 0 0
[junit4:junit4] 2> 974880 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1437911283239223296)]} 0 0
[junit4:junit4] 2> 974881 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1437911283239223296)]} 0 4
[junit4:junit4] 2> 974882 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18]} 0 7
[junit4:junit4] 2> 974884 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1437911283246563328)]} 0 0
[junit4:junit4] 2> 974890 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1437911283249709056)]} 0 0
[junit4:junit4] 2> 974891 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1437911283249709056)]} 0 4
[junit4:junit4] 2> 974895 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1437911283257049088)]} 0 1
[junit4:junit4] 2> 974900 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1437911283260194816)]} 0 0
[junit4:junit4] 2> 974901 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1437911283260194816)]} 0 4
[junit4:junit4] 2> 974904 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1437911283267534848)]} 0 0
[junit4:junit4] 2> 974912 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1437911283272777728)]} 0 0
[junit4:junit4] 2> 974913 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1437911283272777728)]} 0 4
[junit4:junit4] 2> 974914 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21]} 0 7
[junit4:junit4] 2> 974917 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1437911283280117760)]} 0 1
[junit4:junit4] 2> 974924 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1437911283285360640)]} 0 0
[junit4:junit4] 2> 974925 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[22 (1437911283285360640)]} 0 4
[junit4:junit4] 2> 974926 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22]} 0 7
[junit4:junit4] 2> 974928 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1437911283292700672)]} 0 0
[junit4:junit4] 2> 974934 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1437911283295846400)]} 0 0
[junit4:junit4] 2> 974935 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1437911283295846400)]} 0 4
[junit4:junit4] 2> 974938 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1437911283303186432)]} 0 0
[junit4:junit4] 2> 974944 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1437911283306332160)]} 0 0
[junit4:junit4] 2> 974945 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1437911283306332160)]} 0 4
[junit4:junit4] 2> 974948 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1437911283313672192)]} 0 0
[junit4:junit4] 2> 974956 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1437911283318915072)]} 0 0
[junit4:junit4] 2> 974957 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1437911283318915072)]} 0 4
[junit4:junit4] 2> 974958 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25]} 0 7
[junit4:junit4] 2> 974961 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1437911283326255104)]} 0 0
[junit4:junit4] 2> 974967 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1437911283329400832)]} 0 1
[junit4:junit4] 2> 974967 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1437911283329400832)]} 0 4
[junit4:junit4] 2> 974971 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1437911283336740864)]} 0 1
[junit4:junit4] 2> 974977 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1437911283339886592)]} 0 1
[junit4:junit4] 2> 974977 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1437911283339886592)]} 0 4
[junit4:junit4] 2> 974981 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1437911283347226624)]} 0 1
[junit4:junit4] 2> 974986 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1437911283350372352)]} 0 0
[junit4:junit4] 2> 974987 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1437911283350372352)]} 0 4
[junit4:junit4] 2> 974991 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1437911283357712384)]} 0 0
[junit4:junit4] 2> 974997 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1437911283360858112)]} 0 1
[junit4:junit4] 2> 974997 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1437911283360858112)]} 0 4
[junit4:junit4] 2> 975000 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1437911283368198144)]} 0 0
[junit4:junit4] 2> 975008 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1437911283373441024)]} 0 0
[junit4:junit4] 2> 975009 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1437911283373441024)]} 0 4
[junit4:junit4] 2> 975010 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30]} 0 7
[junit4:junit4] 2> 975012 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1437911283380781056)]} 0 0
[junit4:junit4] 2> 975018 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1437911283383926784)]} 0 0
[junit4:junit4] 2> 975019 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1437911283383926784)]} 0 4
[junit4:junit4] 2> 975022 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1437911283391266816)]} 0 0
[junit4:junit4] 2> 975030 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1437911283396509696)]} 0 0
[junit4:junit4] 2> 975031 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[32 (1437911283396509696)]} 0 4
[junit4:junit4] 2> 975032 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32]} 0 7
[junit4:junit4] 2> 975035 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1437911283403849728)]} 0 0
[junit4:junit4] 2> 975042 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1437911283409092608)]} 0 0
[junit4:junit4] 2> 975043 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1437911283409092608)]} 0 4
[junit4:junit4] 2> 975044 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33]} 0 7
[junit4:junit4] 2> 975047 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1437911283416432640)]} 0 1
[junit4:junit4] 2> 975054 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1437911283421675520)]} 0 0
[junit4:junit4] 2> 975055 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[34 (1437911283421675520)]} 0 4
[junit4:junit4] 2> 975056 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34]} 0 7
[junit4:junit4] 2> 975058 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1437911283429015552)]} 0 0
[junit4:junit4] 2> 975064 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1437911283432161280)]} 0 0
[junit4:junit4] 2> 975065 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1437911283432161280)]} 0 4
[junit4:junit4] 2> 975069 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1437911283439501312)]} 0 1
[junit4:junit4] 2> 975076 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1437911283444744192)]} 0 0
[junit4:junit4] 2> 975077 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[36 (1437911283444744192)]} 0 4
[junit4:junit4] 2> 975078 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36]} 0 7
[junit4:junit4] 2> 975080 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1437911283452084224)]} 0 0
[junit4:junit4] 2> 975088 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1437911283457327104)]} 0 0
[junit4:junit4] 2> 975089 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[37 (1437911283457327104)]} 0 4
[junit4:junit4] 2> 975090 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37]} 0 7
[junit4:junit4] 2> 975092 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1437911283464667136)]} 0 0
[junit4:junit4] 2> 975100 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1437911283469910016)]} 0 0
[junit4:junit4] 2> 975101 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[38 (1437911283469910016)]} 0 4
[junit4:junit4] 2> 975102 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38]} 0 7
[junit4:junit4] 2> 975104 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1437911283477250048)]} 0 0
[junit4:junit4] 2> 975110 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1437911283480395776)]} 0 0
[junit4:junit4] 2> 975111 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1437911283480395776)]} 0 4
[junit4:junit4] 2> 975114 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1437911283487735808)]} 0 0
[junit4:junit4] 2> 975122 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1437911283492978688)]} 0 0
[junit4:junit4] 2> 975123 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1437911283492978688)]} 0 4
[junit4:junit4] 2> 975124 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40]} 0 7
[junit4:junit4] 2> 975126 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1437911283500318720)]} 0 0
[junit4:junit4] 2> 975134 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1437911283505561600)]} 0 0
[junit4:junit4] 2> 975135 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[41 (1437911283505561600)]} 0 4
[junit4:junit4] 2> 975136 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41]} 0 7
[junit4:junit4] 2> 975139 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1437911283512901632)]} 0 1
[junit4:junit4] 2> 975146 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1437911283518144512)]} 0 0
[junit4:junit4] 2> 975147 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[42 (1437911283518144512)]} 0 4
[junit4:junit4] 2> 975148 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42]} 0 7
[junit4:junit4] 2> 975150 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1437911283525484544)]} 0 0
[junit4:junit4] 2> 975156 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1437911283528630272)]} 0 0
[junit4:junit4] 2> 975157 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1437911283528630272)]} 0 4
[junit4:junit4] 2> 975160 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1437911283535970304)]} 0 0
[junit4:junit4] 2> 975168 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1437911283541213184)]} 0 0
[junit4:junit4] 2> 975169 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[44 (1437911283541213184)]} 0 4
[junit4:junit4] 2> 975170 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44]} 0 7
[junit4:junit4] 2> 975172 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1437911283548553216)]} 0 0
[junit4:junit4] 2> 975178 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1437911283551698944)]} 0 0
[junit4:junit4] 2> 975179 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1437911283551698944)]} 0 4
[junit4:junit4] 2> 975182 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1437911283559038976)]} 0 0
[junit4:junit4] 2> 975189 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1437911283562184704)]} 0 1
[junit4:junit4] 2> 975189 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1437911283562184704)]} 0 4
[junit4:junit4] 2> 975192 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1437911283569524736)]} 0 0
[junit4:junit4] 2> 975200 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1437911283574767616)]} 0 0
[junit4:junit4] 2> 975201 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1437911283574767616)]} 0 4
[junit4:junit4] 2> 975202 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47]} 0 7
[junit4:junit4] 2> 975205 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1437911283582107648)]} 0 1
[junit4:junit4] 2> 975210 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1437911283585253376)]} 0 0
[junit4:junit4] 2> 975211 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1437911283585253376)]} 0 4
[junit4:junit4] 2> 975215 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1437911283592593408)]} 0 1
[junit4:junit4] 2> 975220 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1437911283596787712)]} 0 0
[junit4:junit4] 2> 975221 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1437911283596787712)]} 0 4
[junit4:junit4] 2> 975225 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1437911283603079168)]} 0 1
[junit4:junit4] 2> 975232 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1437911283608322048)]} 0 0
[junit4:junit4] 2> 975233 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[50 (1437911283608322048)]} 0 4
[junit4:junit4] 2> 975234 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50]} 0 7
[junit4:junit4] 2> 975236 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1437911283615662080)]} 0 0
[junit4:junit4] 2> 975244 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1437911283620904960)]} 0 0
[junit4:junit4] 2> 975245 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[51 (1437911283620904960)]} 0 4
[junit4:junit4] 2> 975246 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51]} 0 7
[junit4:junit4] 2> 975248 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1437911283628244992)]} 0 0
[junit4:junit4] 2> 975256 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1437911283633487872)]} 0 0
[junit4:junit4] 2> 975257 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[52 (1437911283633487872)]} 0 4
[junit4:junit4] 2> 975258 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52]} 0 7
[junit4:junit4] 2> 975260 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1437911283640827904)]} 0 0
[junit4:junit4] 2> 975268 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1437911283646070784)]} 0 0
[junit4:junit4] 2> 975269 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[53 (1437911283646070784)]} 0 4
[junit4:junit4] 2> 975270 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53]} 0 7
[junit4:junit4] 2> 975272 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1437911283653410816)]} 0 0
[junit4:junit4] 2> 975278 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1437911283656556544)]} 0 0
[junit4:junit4] 2> 975279 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1437911283656556544)]} 0 4
[junit4:junit4] 2> 975282 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1437911283663896576)]} 0 0
[junit4:junit4] 2> 975290 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1437911283669139456)]} 0 0
[junit4:junit4] 2> 975291 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1437911283669139456)]} 0 4
[junit4:junit4] 2> 975292 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55]} 0 7
[junit4:junit4] 2> 975295 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1437911283676479488)]} 0 1
[junit4:junit4] 2> 975300 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1437911283679625216)]} 0 0
[junit4:junit4] 2> 975301 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1437911283679625216)]} 0 4
[junit4:junit4] 2> 975304 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1437911283686965248)]} 0 0
[junit4:junit4] 2> 975312 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1437911283692208128)]} 0 0
[junit4:junit4] 2> 975313 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[57 (1437911283692208128)]} 0 4
[junit4:junit4] 2> 975314 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57]} 0 7
[junit4:junit4] 2> 975316 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1437911283699548160)]} 0 0
[junit4:junit4] 2> 975324 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1437911283704791040)]} 0 0
[junit4:junit4] 2> 975325 T1853 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[58 (1437911283704791040)]} 0 4
[junit4:junit4] 2> 975326 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58]} 0 7
[junit4:junit4] 2> 975328 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1437911283712131072)]} 0 0
[junit4:junit4] 2> 975336 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1437911283717373952)]} 0 0
[junit4:junit4] 2> 975337 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[59 (1437911283717373952)]} 0 4
[junit4:junit4] 2> 975338 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59]} 0 7
[junit4:junit4] 2> 975340 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1437911283724713984)]} 0 0
[junit4:junit4] 2> 975346 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1437911283727859712)]} 0 0
[junit4:junit4] 2> 975347 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1437911283727859712)]} 0 4
[junit4:junit4] 2> 975352 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1437911283736248320)]} 0 1
[junit4:junit4] 2> 975359 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1437911283741491200)]} 0 0
[junit4:junit4] 2> 975360 T1839 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[61 (1437911283741491200)]} 0 4
[junit4:junit4] 2> 975361 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61]} 0 7
[junit4:junit4] 2> 975363 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1437911283748831232)]} 0 0
[junit4:junit4] 2> 975369 T1884 C715 P10317 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:62105/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1437911283751976960)]} 0 0
[junit4:junit4] 2> 975370 T1855 C716 P62105 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1437911283751976960)]} 0 4
[junit4:junit4] 2> 975374 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1437911283759316992)]} 0 0
[junit4:junit4] 2> 975379 T1865 C713 P41605 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:20986/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1437911283762462720)]} 0 0
[junit4:junit4] 2> 975380 T1837 C714 P20986 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1437911283762462720)]} 0 4
[junit4:junit4] 2> 975383 T1813 C717 P11430 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1437911283769802752)]} 0 0
[junit4:junit4] 2> 975392 T1865 C713 P
[...truncated too long message...]
updating... (live nodes size: 6)
[junit4:junit4] 2> 1402410 T2155 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1402410 T2155 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 1402410 T2155 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1402410 T2155 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1405637 T2212 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:23965 couldn't connect to http://127.0.0.1:39884/collection1/, counting as success
[junit4:junit4] 2> 1405637 T2212 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:23965 DONE. sync succeeded
[junit4:junit4] 2> 1405638 T2212 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 1405638 T2212 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4:junit4] 2> 1405639 T2212 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1e4a067e
[junit4:junit4] 2> 1405640 T2212 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=941,cumulative_deletesById=476,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 1405641 T2212 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1405641 T2212 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1405641 T2212 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1405644 T2212 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1405645 T2212 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1405645 T2212 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12;done=false>>]
[junit4:junit4] 2> 1405646 T2212 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12
[junit4:junit4] 2> 1405646 T2212 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12/index;done=false>>]
[junit4:junit4] 2> 1405646 T2212 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371299229671/jetty12/index
[junit4:junit4] 2> 1405646 T2212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1405647 T2212 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1405647 T2212 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1405647 T2212 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=in, timezone=US/Eastern
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=241443848,total=529530880
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestNumberUtils, NoCacheHeaderTest, TestQuerySenderNoQuery, TestSuggestSpellingConverter, HighlighterConfigTest, TestDFRSimilarityFactory, LeaderElectionIntegrationTest, HardAutoCommitTest, RequestHandlersTest, SolrPluginUtilsTest, TestUtils, TestSolrQueryParserResource, UpdateParamsTest, NotRequiredUniqueKeyTest, IndexSchemaRuntimeFieldTest, DirectUpdateHandlerTest, OpenExchangeRatesOrgProviderTest, TestComponentsName, StandardRequestHandlerTest, TestRandomFaceting, TestElisionMultitermQuery, TestDistributedGrouping, ShardRoutingCustomTest, TestIndexingPerformance, BasicDistributedZk2Test, SimplePostToolTest, TestSolrQueryParserDefaultOperatorResource, TermVectorComponentDistributedTest, PreAnalyzedUpdateProcessorTest, CopyFieldTest, RequiredFieldsTest, TestPropInject, CollectionsAPIDistributedZkTest, TestDynamicFieldCollectionResource, SolrCoreCheckLockOnStartupTest, TestPostingsSolrHighlighter, TestHashPartitioner, TestFieldResource, TestFastOutputStream, TestMergePolicyConfig, DocumentAnalysisRequestHandlerTest, DebugComponentTest, SuggesterTSTTest, TestLazyCores, TestDistributedSearch, XsltUpdateRequestHandlerTest, TestAtomicUpdateErrorCases, TestCodecSupport, DOMUtilTest, TestQueryTypes, BasicZkTest, ChaosMonkeyShardSplitTest, TestJmxIntegration, XmlUpdateRequestHandlerTest, PingRequestHandlerTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 261.77s, 1 test, 1 failure <<< FAILURES!
[...truncated 518 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:386: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:366: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:887: There were test failures: 296 suites, 1243 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 54 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4067 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4067/
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 22 from http://127.0.0.1:15122/collection1lastClient and got 3 from http://127.0.0.1:15139/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 22 from http://127.0.0.1:15122/collection1lastClient and got 3 from http://127.0.0.1:15139/collection1
at __randomizedtesting.SeedInfo.seed([45EBF799E0CF3878:C40D798197905844]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<209> but was:<69>
Stack Trace:
java.lang.AssertionError: expected:<209> but was:<69>
at __randomizedtesting.SeedInfo.seed([45EBF799E0CF3878:C40D798197905844]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9401 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 322671 T836 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 322679 T836 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371269417899
[junit4:junit4] 2> 322681 T836 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 322682 T837 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 322784 T836 oasc.ZkTestServer.run start zk server on port:42111
[junit4:junit4] 2> 322785 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 322795 T843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1173d0c5 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 322795 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 322795 T836 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 322813 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 322820 T845 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@238d4b23 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 322820 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 322821 T836 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 322837 T836 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 322845 T836 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 322848 T836 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 322851 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 322852 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 322864 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 322864 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 322970 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 322971 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 322974 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 322975 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 322986 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 322987 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 322990 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 322991 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 322994 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 322995 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 322998 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 322999 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 323002 T836 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 323002 T836 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 323291 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 323296 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16507
[junit4:junit4] 2> 323297 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 323298 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 323298 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231
[junit4:junit4] 2> 323299 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/solr.xml
[junit4:junit4] 2> 323299 T836 oasc.CoreContainer.<init> New CoreContainer 959506201
[junit4:junit4] 2> 323300 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/'
[junit4:junit4] 2> 323300 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/'
[junit4:junit4] 2> 323435 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 323436 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 323436 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 323437 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 323437 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 323438 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 323438 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 323439 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 323439 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 323440 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 323460 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 323461 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 323462 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 323463 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 323466 T856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@430df842 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 323467 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 323484 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 323499 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 323501 T858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f94d995 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 323501 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 323508 T836 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 323512 T836 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 323521 T836 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 323523 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16507_
[junit4:junit4] 2> 323525 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16507_
[junit4:junit4] 2> 323535 T836 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 323547 T836 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 323554 T836 oasc.Overseer.start Overseer (id=89867512571953155-127.0.0.1:16507_-n_0000000000) starting
[junit4:junit4] 2> 323559 T836 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 323570 T860 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 323571 T836 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 323573 T836 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 323575 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 323577 T859 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 323580 T861 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 323580 T861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 325081 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 325082 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:16507_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:16507"}
[junit4:junit4] 2> 325083 T859 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 325083 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 325105 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 325584 T861 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1
[junit4:junit4] 2> 325584 T861 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 325585 T861 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 325585 T861 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 325587 T861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/'
[junit4:junit4] 2> 325590 T861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 325590 T861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/lib/README' to classloader
[junit4:junit4] 2> 325653 T861 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 325723 T861 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 325826 T861 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 325831 T861 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 326410 T861 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 326424 T861 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 326428 T861 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 326442 T861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 326447 T861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 326452 T861 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 326453 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 326453 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 326453 T861 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 326454 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 326455 T861 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 326455 T861 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 326518 T861 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371269418231/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/
[junit4:junit4] 2> 326519 T861 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 326519 T861 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 326521 T861 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data
[junit4:junit4] 2> 326522 T861 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index/
[junit4:junit4] 2> 326523 T861 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 326524 T861 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index
[junit4:junit4] 2> 326547 T861 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 326548 T861 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 326550 T861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 326551 T861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 326551 T861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 326552 T861 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 326553 T861 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 326553 T861 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 326571 T861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 326585 T861 oass.SolrIndexSearcher.<init> Opening Searcher@4b8d9424 main
[junit4:junit4] 2> 326587 T861 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 326587 T861 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 326591 T862 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b8d9424 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 326593 T861 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 326593 T861 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16507 collection:control_collection shard:shard1
[junit4:junit4] 2> 326594 T861 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 326612 T861 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 326615 T861 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 326615 T861 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 326616 T861 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16507/collection1/
[junit4:junit4] 2> 326616 T861 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 326616 T861 oasc.SyncStrategy.syncToMe http://127.0.0.1:16507/collection1/ has no replicas
[junit4:junit4] 2> 326616 T861 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16507/collection1/
[junit4:junit4] 2> 326617 T861 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 328113 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 328130 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 328175 T861 oasc.ZkController.register We are http://127.0.0.1:16507/collection1/ and leader is http://127.0.0.1:16507/collection1/
[junit4:junit4] 2> 328175 T861 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16507
[junit4:junit4] 2> 328175 T861 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 328175 T861 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 328175 T861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 328177 T861 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 328178 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 328178 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 328179 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 328191 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 328192 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 328194 T865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@628441a7 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 328194 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 328196 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 328198 T836 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 328425 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 328427 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:53832
[junit4:junit4] 2> 328428 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 328428 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 328428 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416
[junit4:junit4] 2> 328429 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/solr.xml
[junit4:junit4] 2> 328429 T836 oasc.CoreContainer.<init> New CoreContainer 233765018
[junit4:junit4] 2> 328429 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/'
[junit4:junit4] 2> 328430 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/'
[junit4:junit4] 2> 328518 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 328519 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 328519 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 328519 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 328520 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 328520 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 328520 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 328521 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 328521 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 328521 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 328533 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 328534 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 328534 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 328535 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 328537 T876 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@780d5011 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 328538 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 328540 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 328548 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 328551 T878 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e91bdf8 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 328551 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 328564 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 329570 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53832_
[junit4:junit4] 2> 329576 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53832_
[junit4:junit4] 2> 329581 T865 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 329581 T878 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 329581 T858 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 329581 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 329595 T879 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 329596 T879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 329645 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 329666 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:16507__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:16507_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:16507"}
[junit4:junit4] 2> 329671 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53832_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53832"}
[junit4:junit4] 2> 329671 T859 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 329671 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 329691 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 329691 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 329691 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 330599 T879 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1
[junit4:junit4] 2> 330599 T879 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 330600 T879 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 330600 T879 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 330602 T879 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/'
[junit4:junit4] 2> 330603 T879 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 330603 T879 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/lib/README' to classloader
[junit4:junit4] 2> 330644 T879 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 330692 T879 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 330793 T879 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 330798 T879 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 331300 T879 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 331310 T879 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 331313 T879 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 331327 T879 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 331331 T879 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 331335 T879 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 331336 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 331337 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 331337 T879 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 331338 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 331338 T879 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 331339 T879 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 331339 T879 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371269423416/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/
[junit4:junit4] 2> 331339 T879 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 331340 T879 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 331340 T879 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1
[junit4:junit4] 2> 331341 T879 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index/
[junit4:junit4] 2> 331342 T879 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 331343 T879 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index
[junit4:junit4] 2> 331347 T879 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 331348 T879 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 331350 T879 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 331350 T879 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 331351 T879 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 331351 T879 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 331352 T879 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 331352 T879 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 331352 T879 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 331353 T879 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 331353 T879 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 331367 T879 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 331374 T879 oass.SolrIndexSearcher.<init> Opening Searcher@4194c10f main
[junit4:junit4] 2> 331375 T879 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 331375 T879 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 331379 T880 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4194c10f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 331380 T879 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 331380 T879 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53832 collection:collection1 shard:shard1
[junit4:junit4] 2> 331381 T879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 331396 T879 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 331405 T879 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 331405 T879 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 331406 T879 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 331406 T879 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 331406 T879 oasc.SyncStrategy.syncToMe http://127.0.0.1:53832/collection1/ has no replicas
[junit4:junit4] 2> 331406 T879 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 331406 T879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 332700 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 332717 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 332717 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 332717 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 332755 T879 oasc.ZkController.register We are http://127.0.0.1:53832/collection1/ and leader is http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 332755 T879 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53832
[junit4:junit4] 2> 332755 T879 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 332755 T879 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 332755 T879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 332758 T879 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 332759 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 332759 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 332760 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 332951 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 332954 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15080
[junit4:junit4] 2> 332954 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 332955 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 332955 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989
[junit4:junit4] 2> 332955 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/solr.xml
[junit4:junit4] 2> 332956 T836 oasc.CoreContainer.<init> New CoreContainer 277165989
[junit4:junit4] 2> 332956 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/'
[junit4:junit4] 2> 332956 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/'
[junit4:junit4] 2> 333041 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 333042 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 333043 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 333043 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 333043 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 333044 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 333044 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 333044 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 333045 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 333045 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 333056 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 333057 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 333057 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 333058 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 333060 T892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a7b1e0 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 333061 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 333072 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 333085 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 333087 T894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36df849d name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 333087 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 333105 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 334110 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15080_
[junit4:junit4] 2> 334126 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15080_
[junit4:junit4] 2> 334209 T878 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T858 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T894 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T865 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 334209 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334209 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334210 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334282 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 334283 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53832__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53832_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53832"}
[junit4:junit4] 2> 334285 T895 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 334285 T895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 334351 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15080_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15080"}
[junit4:junit4] 2> 334352 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 334352 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 334375 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334376 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334375 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 334376 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 335331 T895 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1
[junit4:junit4] 2> 335331 T895 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 335332 T895 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 335332 T895 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 335333 T895 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/'
[junit4:junit4] 2> 335334 T895 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/lib/README' to classloader
[junit4:junit4] 2> 335335 T895 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 335370 T895 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 335411 T895 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 335512 T895 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 335516 T895 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 335935 T895 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 335944 T895 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 335946 T895 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 335957 T895 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 335961 T895 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 335964 T895 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 335965 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 335966 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 335967 T895 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 335967 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 335967 T895 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 335968 T895 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 335968 T895 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371269427989/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/
[junit4:junit4] 2> 335968 T895 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 335969 T895 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 335970 T895 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2
[junit4:junit4] 2> 335970 T895 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index/
[junit4:junit4] 2> 335971 T895 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 335971 T895 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index
[junit4:junit4] 2> 335988 T895 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 335989 T895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 335991 T895 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 335991 T895 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 335992 T895 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 335992 T895 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 335993 T895 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 335993 T895 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 335993 T895 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 335994 T895 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 335994 T895 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 336005 T895 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 336011 T895 oass.SolrIndexSearcher.<init> Opening Searcher@3690524a main
[junit4:junit4] 2> 336012 T895 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 336012 T895 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 336016 T896 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3690524a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 336017 T895 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 336017 T895 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15080 collection:collection1 shard:shard2
[junit4:junit4] 2> 336018 T895 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 336031 T895 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 336040 T895 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 336040 T895 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 336040 T895 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 336040 T895 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 336040 T895 oasc.SyncStrategy.syncToMe http://127.0.0.1:15080/collection1/ has no replicas
[junit4:junit4] 2> 336041 T895 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 336041 T895 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 337390 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 337407 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337407 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337407 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337407 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 337434 T895 oasc.ZkController.register We are http://127.0.0.1:15080/collection1/ and leader is http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 337434 T895 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15080
[junit4:junit4] 2> 337434 T895 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 337434 T895 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 337435 T895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 337437 T895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 337437 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 337438 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 337438 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 337629 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 337632 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15100
[junit4:junit4] 2> 337632 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 337633 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 337633 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667
[junit4:junit4] 2> 337633 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/solr.xml
[junit4:junit4] 2> 337634 T836 oasc.CoreContainer.<init> New CoreContainer 230704970
[junit4:junit4] 2> 337634 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/'
[junit4:junit4] 2> 337635 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/'
[junit4:junit4] 2> 337734 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 337734 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 337735 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 337735 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 337735 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 337736 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 337736 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 337736 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 337737 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 337737 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 337748 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 337749 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 337749 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 337750 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 337753 T908 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25eb5807 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 337753 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 337765 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 337773 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 337775 T910 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e554da4 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 337776 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 337787 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 338790 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15100_
[junit4:junit4] 2> 338802 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15100_
[junit4:junit4] 2> 338806 T865 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T910 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T878 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338806 T894 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338806 T858 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 338807 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338807 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338807 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338814 T911 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 338814 T911 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 338922 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 338923 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15080__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15080_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15080"}
[junit4:junit4] 2> 338927 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15100_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15100"}
[junit4:junit4] 2> 338927 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 338927 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 338992 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 338992 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 339818 T911 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1
[junit4:junit4] 2> 339818 T911 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 339819 T911 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 339819 T911 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 339821 T911 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/'
[junit4:junit4] 2> 339823 T911 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 339823 T911 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/lib/README' to classloader
[junit4:junit4] 2> 339873 T911 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 339929 T911 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 340030 T911 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 340037 T911 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 340622 T911 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 340631 T911 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 340633 T911 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 340644 T911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 340648 T911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 340651 T911 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 340652 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 340653 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 340653 T911 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 340654 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 340654 T911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 340654 T911 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 340655 T911 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371269432667/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/
[junit4:junit4] 2> 340655 T911 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 340655 T911 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 340656 T911 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3
[junit4:junit4] 2> 340657 T911 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index/
[junit4:junit4] 2> 340657 T911 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 340658 T911 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index
[junit4:junit4] 2> 340687 T911 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 340687 T911 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 340690 T911 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 340690 T911 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 340690 T911 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 340691 T911 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 340691 T911 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 340692 T911 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 340692 T911 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 340692 T911 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 340693 T911 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 340703 T911 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 340710 T911 oass.SolrIndexSearcher.<init> Opening Searcher@763a2bfc main
[junit4:junit4] 2> 340710 T911 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 340711 T911 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 340714 T912 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@763a2bfc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 340715 T911 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 340715 T911 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15100 collection:collection1 shard:shard3
[junit4:junit4] 2> 340716 T911 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 340730 T911 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 340741 T911 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 340741 T911 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 340742 T911 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 340742 T911 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 340742 T911 oasc.SyncStrategy.syncToMe http://127.0.0.1:15100/collection1/ has no replicas
[junit4:junit4] 2> 340742 T911 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 340742 T911 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 342027 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 342071 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342072 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342072 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342071 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342071 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342093 T911 oasc.ZkController.register We are http://127.0.0.1:15100/collection1/ and leader is http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 342093 T911 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15100
[junit4:junit4] 2> 342093 T911 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 342093 T911 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 342094 T911 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 342095 T911 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 342097 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 342098 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 342098 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 342323 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 342327 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15103
[junit4:junit4] 2> 342327 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 342327 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 342328 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328
[junit4:junit4] 2> 342328 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/solr.xml
[junit4:junit4] 2> 342328 T836 oasc.CoreContainer.<init> New CoreContainer 1197773348
[junit4:junit4] 2> 342329 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/'
[junit4:junit4] 2> 342329 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/'
[junit4:junit4] 2> 342414 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 342414 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 342415 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 342415 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 342415 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 342416 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 342416 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 342416 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 342417 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 342417 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 342429 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 342430 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 342430 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 342431 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 342434 T924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@321d96f7 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 342434 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 342437 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 342451 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 342455 T926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c341ed4 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 342456 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 342475 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 343479 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15103_
[junit4:junit4] 2> 343507 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15103_
[junit4:junit4] 2> 343511 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 343512 T894 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T858 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T865 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T926 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343512 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343512 T878 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343513 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343512 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343514 T910 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 343513 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343525 T927 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 343525 T927 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 343586 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 343587 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15100__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15100_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15100"}
[junit4:junit4] 2> 343591 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15103_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15103"}
[junit4:junit4] 2> 343591 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 343592 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 343603 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 343603 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 344532 T927 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1
[junit4:junit4] 2> 344532 T927 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 344533 T927 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 344533 T927 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 344535 T927 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/'
[junit4:junit4] 2> 344537 T927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 344537 T927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/lib/README' to classloader
[junit4:junit4] 2> 344591 T927 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 344667 T927 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 344768 T927 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 344775 T927 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 345388 T927 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 345400 T927 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 345405 T927 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 345420 T927 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 345425 T927 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 345430 T927 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 345431 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 345432 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 345433 T927 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 345434 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 345434 T927 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 345434 T927 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 345435 T927 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371269437328/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/
[junit4:junit4] 2> 345435 T927 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 345436 T927 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 345437 T927 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4
[junit4:junit4] 2> 345438 T927 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index/
[junit4:junit4] 2> 345439 T927 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 345439 T927 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index
[junit4:junit4] 2> 345458 T927 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 345459 T927 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 345462 T927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 345462 T927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 345463 T927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 345464 T927 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 345465 T927 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 345465 T927 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 345465 T927 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 345466 T927 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 345466 T927 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 345484 T927 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 345495 T927 oass.SolrIndexSearcher.<init> Opening Searcher@2842f8db main
[junit4:junit4] 2> 345499 T927 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 345499 T927 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 345505 T928 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2842f8db main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 345507 T927 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 345507 T927 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15103 collection:collection1 shard:shard1
[junit4:junit4] 2> 345544 T927 oasc.ZkController.register We are http://127.0.0.1:15103/collection1/ and leader is http://127.0.0.1:53832/collection1/
[junit4:junit4] 2> 345544 T927 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15103
[junit4:junit4] 2> 345545 T927 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 345545 T927 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C195 name=collection1 org.apache.solr.core.SolrCore@15c25758 url=http://127.0.0.1:15103/collection1 node=127.0.0.1:15103_ C195_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:15103_, base_url=http://127.0.0.1:15103}
[junit4:junit4] 2> 345546 T929 C195 P15103 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 345546 T927 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 345546 T929 C195 P15103 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 345547 T929 C195 P15103 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 345547 T929 C195 P15103 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 345548 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 345548 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 345549 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 345549 T929 C195 P15103 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 345577 T868 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:15103__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 345846 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 345849 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15108
[junit4:junit4] 2> 345850 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 345851 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 345851 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792
[junit4:junit4] 2> 345852 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/solr.xml
[junit4:junit4] 2> 345852 T836 oasc.CoreContainer.<init> New CoreContainer 1379866054
[junit4:junit4] 2> 345853 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/'
[junit4:junit4] 2> 345853 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/'
[junit4:junit4] 2> 345983 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 345983 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 345984 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 345984 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 345985 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 345985 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 345986 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 345986 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 345987 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 345988 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 346005 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 346006 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 346006 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 346007 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 346012 T941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d245a44 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 346013 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 346022 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 346035 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 346037 T943 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a1cfe8e name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 346038 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 346049 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 346620 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 346621 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15103__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15103_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15103"}
[junit4:junit4] 2> 346642 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346642 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347054 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15108_
[junit4:junit4] 2> 347057 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15108_
[junit4:junit4] 2> 347059 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347060 T894 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T858 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T865 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T878 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347060 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347060 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347060 T926 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347060 T943 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347062 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347061 T910 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 347061 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347062 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 347073 T944 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 347074 T944 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 347580 T868 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:15103__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 347580 T868 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:15103__collection1&state=recovering&nodeName=127.0.0.1:15103_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 348149 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 348149 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15108_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15108"}
[junit4:junit4] 2> 348150 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 348150 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 348179 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 348179 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 349079 T944 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1
[junit4:junit4] 2> 349079 T944 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 349080 T944 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 349080 T944 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 349082 T944 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/'
[junit4:junit4] 2> 349084 T944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/lib/README' to classloader
[junit4:junit4] 2> 349084 T944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 349136 T944 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 349198 T944 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 349200 T944 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 349206 T944 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C196 name=collection1 org.apache.solr.core.SolrCore@15c25758 url=http://127.0.0.1:15103/collection1 node=127.0.0.1:15103_ C196_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:15103_, base_url=http://127.0.0.1:15103}
[junit4:junit4] 2> 349583 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53832/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 349584 T929 C196 P15103 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:15103 START replicas=[http://127.0.0.1:53832/collection1/] nUpdates=100
[junit4:junit4] 2> 349586 T929 C196 P15103 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 349589 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 349589 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 349590 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 349590 T929 C196 P15103 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 349591 T929 C196 P15103 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53832/collection1/. core=collection1
[junit4:junit4] 2> 349591 T929 C196 P15103 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C197 name=collection1 org.apache.solr.core.SolrCore@4d5898ce url=http://127.0.0.1:53832/collection1 node=127.0.0.1:53832_ C197_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53832_, base_url=http://127.0.0.1:53832, leader=true}
[junit4:junit4] 2> 349617 T869 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 349620 T870 C197 P53832 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 349635 T870 C197 P53832 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 349639 T870 C197 P53832 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 349649 T870 C197 P53832 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 349650 T870 C197 P53832 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 349652 T870 C197 P53832 oass.SolrIndexSearcher.<init> Opening Searcher@7c7284f4 realtime
[junit4:junit4] 2> 349652 T870 C197 P53832 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 349653 T870 C197 P53832 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 35
[junit4:junit4] 2> 349654 T929 C196 P15103 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 349654 T929 C196 P15103 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 349658 T871 C197 P53832 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 349659 T871 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
[junit4:junit4] 2> 349660 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 349660 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 349660 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 349664 T871 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 349666 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 349668 T929 C196 P15103 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884
[junit4:junit4] 2> 349669 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfe4b75 fullCopy=false
[junit4:junit4] 2> 349674 T871 C197 P53832 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 349677 T929 C196 P15103 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 349745 T929 C196 P15103 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 349746 T929 C196 P15103 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 349752 T929 C196 P15103 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 349754 T929 C196 P15103 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 349755 T929 C196 P15103 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 349756 T929 C196 P15103 oass.SolrIndexSearcher.<init> Opening Searcher@38032b2b main
[junit4:junit4] 2> 349758 T928 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38032b2b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 349758 T929 C196 P15103 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884;done=true>>]
[junit4:junit4] 2> 349759 T929 C196 P15103 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884
[junit4:junit4] 2> 349759 T929 C196 P15103 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty4/index.20130615061044884
[junit4:junit4] 2> 349760 T929 C196 P15103 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 349760 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 349761 T929 C196 P15103 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 349761 T929 C196 P15103 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 349763 T929 C196 P15103 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 349953 T944 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 349965 T944 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 349969 T944 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 349984 T944 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 350003 T944 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 350007 T944 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 350008 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 350009 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 350009 T944 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 350010 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 350011 T944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 350011 T944 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 350011 T944 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371269440792/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/
[junit4:junit4] 2> 350012 T944 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 350012 T944 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 350013 T944 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5
[junit4:junit4] 2> 350014 T944 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index/
[junit4:junit4] 2> 350015 T944 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 350016 T944 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index
[junit4:junit4] 2> 350021 T944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 350022 T944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 350025 T944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 350025 T944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 350026 T944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 350026 T944 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 350027 T944 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 350027 T944 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 350031 T944 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 350031 T944 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 350032 T944 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 350048 T944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 350058 T944 oass.SolrIndexSearcher.<init> Opening Searcher@3b890dc0 main
[junit4:junit4] 2> 350059 T944 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 350059 T944 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 350064 T947 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b890dc0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 350066 T944 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 350066 T944 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15108 collection:collection1 shard:shard2
[junit4:junit4] 2> 350070 T944 oasc.ZkController.register We are http://127.0.0.1:15108/collection1/ and leader is http://127.0.0.1:15080/collection1/
[junit4:junit4] 2> 350070 T944 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15108
[junit4:junit4] 2> 350071 T944 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 350071 T944 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C198 name=collection1 org.apache.solr.core.SolrCore@33a2c26b url=http://127.0.0.1:15108/collection1 node=127.0.0.1:15108_ C198_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:15108_, base_url=http://127.0.0.1:15108}
[junit4:junit4] 2> 350071 T948 C198 P15108 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 350072 T948 C198 P15108 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 350072 T944 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 350072 T948 C198 P15108 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 350073 T948 C198 P15108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 350073 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 350074 T948 C198 P15108 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 350074 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 350075 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 350099 T884 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:15108__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 350368 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 350373 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15115
[junit4:junit4] 2> 350373 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 350374 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 350375 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315
[junit4:junit4] 2> 350375 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/solr.xml
[junit4:junit4] 2> 350376 T836 oasc.CoreContainer.<init> New CoreContainer 1431465309
[junit4:junit4] 2> 350376 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/'
[junit4:junit4] 2> 350377 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/'
[junit4:junit4] 2> 350509 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 350510 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 350511 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 350511 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 350512 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 350512 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 350513 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 350513 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 350514 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 350514 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 350532 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 350533 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 350533 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 350534 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 350537 T960 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37189f9d name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 350538 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 350550 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 350562 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 350564 T962 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bd803 name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 350564 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 350575 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 351194 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 351195 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15103__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15103_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15103"}
[junit4:junit4] 2> 351226 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15108__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15108_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15108"}
[junit4:junit4] 2> 351248 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351248 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351579 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15115_
[junit4:junit4] 2> 351582 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15115_
[junit4:junit4] 2> 351584 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 351585 T926 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T894 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T943 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T962 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351585 T858 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T878 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351585 T865 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351586 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351586 T910 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 351586 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351586 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351585 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351585 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351587 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 351599 T963 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 351600 T963 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 352102 T884 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:15108__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 352102 T884 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:15108__collection1&state=recovering&nodeName=127.0.0.1:15108_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 352756 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 352757 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15115_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15115"}
[junit4:junit4] 2> 352757 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 352757 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 352772 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 352773 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 353605 T963 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1
[junit4:junit4] 2> 353605 T963 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 353606 T963 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 353606 T963 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 353608 T963 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/'
[junit4:junit4] 2> 353609 T963 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/lib/README' to classloader
[junit4:junit4] 2> 353609 T963 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 353646 T963 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 353689 T963 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 353690 T963 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 353695 T963 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C199 name=collection1 org.apache.solr.core.SolrCore@33a2c26b url=http://127.0.0.1:15108/collection1 node=127.0.0.1:15108_ C199_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:15108_, base_url=http://127.0.0.1:15108}
[junit4:junit4] 2> 354104 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:15080/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 354104 T948 C199 P15108 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:15108 START replicas=[http://127.0.0.1:15080/collection1/] nUpdates=100
[junit4:junit4] 2> 354105 T948 C199 P15108 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 354105 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 354105 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 354105 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 354106 T948 C199 P15108 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 354106 T948 C199 P15108 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:15080/collection1/. core=collection1
[junit4:junit4] 2> 354106 T948 C199 P15108 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C200 name=collection1 org.apache.solr.core.SolrCore@513b1428 url=http://127.0.0.1:15080/collection1 node=127.0.0.1:15080_ C200_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:15080_, base_url=http://127.0.0.1:15080, leader=true}
[junit4:junit4] 2> 354115 T885 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 354119 T886 C200 P15080 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 354125 T886 C200 P15080 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 354125 T886 C200 P15080 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 354171 T886 C200 P15080 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 354172 T886 C200 P15080 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 354174 T886 C200 P15080 oass.SolrIndexSearcher.<init> Opening Searcher@7ccd6ee1 realtime
[junit4:junit4] 2> 354175 T886 C200 P15080 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 354175 T886 C200 P15080 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 56
[junit4:junit4] 2> 354176 T948 C199 P15108 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 354177 T948 C199 P15108 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 354179 T887 C200 P15080 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 354180 T887 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 354180 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 354181 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 354181 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 354184 T887 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 354185 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 354187 T948 C199 P15108 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404
[junit4:junit4] 2> 354188 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5350c62a fullCopy=false
[junit4:junit4] 2> 354192 T887 C200 P15080 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 354194 T948 C199 P15108 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 354197 T963 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 354210 T963 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 354214 T963 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 354222 T948 C199 P15108 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 354222 T948 C199 P15108 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 354227 T948 C199 P15108 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 354228 T948 C199 P15108 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 354229 T948 C199 P15108 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 354229 T963 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 354230 T948 C199 P15108 oass.SolrIndexSearcher.<init> Opening Searcher@193d71d4 main
[junit4:junit4] 2> 354231 T947 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@193d71d4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 354232 T948 C199 P15108 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404;done=true>>]
[junit4:junit4] 2> 354232 T948 C199 P15108 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404
[junit4:junit4] 2> 354232 T948 C199 P15108 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty5/index.20130615061049404
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 354234 T948 C199 P15108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 354236 T948 C199 P15108 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 354255 T963 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 354262 T963 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 354264 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 354264 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 354265 T963 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 354266 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 354266 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 354266 T963 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 354267 T963 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371269445315/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/
[junit4:junit4] 2> 354267 T963 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45cd286a
[junit4:junit4] 2> 354268 T963 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 354269 T963 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6
[junit4:junit4] 2> 354270 T963 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index/
[junit4:junit4] 2> 354271 T963 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 354272 T963 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index
[junit4:junit4] 2> 354278 T963 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371269417898/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 354278 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 354278 T963 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 354279 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15108__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15108_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15108"}
[junit4:junit4] 2> 354282 T963 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 354282 T963 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 354283 T963 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 354285 T963 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 354286 T963 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 354286 T963 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 354290 T963 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 354290 T963 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 354291 T963 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 354299 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354300 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354299 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 354308 T963 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 354317 T963 oass.SolrIndexSearcher.<init> Opening Searcher@68058dec main
[junit4:junit4] 2> 354319 T963 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 354319 T963 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 354324 T966 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@68058dec main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 354326 T963 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 354327 T963 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:15115 collection:collection1 shard:shard3
[junit4:junit4] 2> 354331 T963 oasc.ZkController.register We are http://127.0.0.1:15115/collection1/ and leader is http://127.0.0.1:15100/collection1/
[junit4:junit4] 2> 354332 T963 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15115
[junit4:junit4] 2> 354332 T963 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 354332 T963 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C201 name=collection1 org.apache.solr.core.SolrCore@13e227a5 url=http://127.0.0.1:15115/collection1 node=127.0.0.1:15115_ C201_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:15115_, base_url=http://127.0.0.1:15115}
[junit4:junit4] 2> 354333 T967 C201 P15115 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 354333 T963 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 354333 T967 C201 P15115 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 354334 T967 C201 P15115 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 354334 T967 C201 P15115 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 354335 T836 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4:junit4] 2> 354335 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 354336 T967 C201 P15115 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 354336 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 354356 T900 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:15115__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 354624 T836 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 354628 T836 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15122
[junit4:junit4] 2> 354629 T836 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 354629 T836 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 354630 T836 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572
[junit4:junit4] 2> 354630 T836 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/solr.xml
[junit4:junit4] 2> 354631 T836 oasc.CoreContainer.<init> New CoreContainer 1851212974
[junit4:junit4] 2> 354631 T836 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/'
[junit4:junit4] 2> 354632 T836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/'
[junit4:junit4] 2> 354762 T836 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 354763 T836 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 354763 T836 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 354764 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 354764 T836 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 354765 T836 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 354766 T836 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 354766 T836 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 354767 T836 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 354767 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 354783 T836 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 354784 T836 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42111/solr
[junit4:junit4] 2> 354784 T836 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 354785 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 354808 T979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42f91b74 name:ZooKeeperConnection Watcher:127.0.0.1:42111 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 354809 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 354812 T836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 354823 T836 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 354825 T981 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a7367f name:ZooKeeperConnection Watcher:127.0.0.1:42111/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 354825 T836 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 354844 T836 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 355804 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 355805 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:15115__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15115_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15115"}
[junit4:junit4] 2> 355844 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355844 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355848 T836 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15122_
[junit4:junit4] 2> 355850 T836 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15122_
[junit4:junit4] 2> 355858 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355858 T926 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T943 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T858 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T962 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T894 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 355859 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355859 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355859 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355858 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355858 T878 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T865 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355858 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355861 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355860 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 355860 T910 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355861 T981 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 355866 T982 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 355866 T982 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 356361 T900 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:15115__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 356361 T900 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:15115__collection1&state=recovering&nodeName=127.0.0.1:15115_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2005
[junit4:junit4] 2> 357352 T859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 357353 T859 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:15122_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:15122"}
[junit4:junit4] 2> 357353 T859 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 357353 T859 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 357369 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357370 T962 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T878 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T981 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T865 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357369 T858 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 357870 T982 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1
[junit4:junit4] 2> 357870 T982 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 357871 T982 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 357871 T982 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 357873 T982 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1/'
[junit4:junit4] 2> 357875 T982 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 357875 T982 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371269449572/collection1/lib/README' to classloader
[junit4:junit4] 2> 357930 T982 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 357990 T982 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2
[...truncated too long message...]
617 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:20586 DONE. sync succeeded
[junit4:junit4] 2> 1546434 T2617 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 1546434 T2617 oasc.ShardLeaderElectionContext.runLeaderProcess Sync was not a success but no one else is active! I am the leader
[junit4:junit4] 2> 1546434 T2617 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:20586/collection1/
[junit4:junit4] 2> 1546435 T2617 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@408211a1
[junit4:junit4] 2> 1546442 T2617 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=85,cumulative_deletesById=30,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 1546442 T2617 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1546442 T2617 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1546443 T2617 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1546470 T2617 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2/index;done=false>>]
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2/index
[junit4:junit4] 2> 1546471 T2617 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2;done=false>>]
[junit4:junit4] 2> 1546472 T2617 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371270497767/jetty2
[junit4:junit4] 2> 1546472 T2617 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1546472 T2617 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/89867583342182410-127.0.0.1:20586__collection1-n_0000000002
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:260)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1546473 T2617 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1546473 T2617 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1546474 T2617 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1546474 T2617 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=SimpleText), text=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=SimpleText), rnd_b=PostingsFormat(name=Direct), intDefault=PostingsFormat(name=SimpleText), timestamp=PostingsFormat(name=SimpleText), id=PostingsFormat(name=Direct), a_t=PostingsFormat(name=SimpleText), range_facet_sl=PostingsFormat(name=MockRandom), range_facet_si=MockFixedIntBlock(blockSize=325), other_tl1=PostingsFormat(name=SimpleText), multiDefault=MockFixedIntBlock(blockSize=325), a_si=MockFixedIntBlock(blockSize=325)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=nl_NL, timezone=Australia/Broken_Hill
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=152398288,total=321912832
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestStressRecovery, TestFastLRUCache, WordBreakSolrSpellCheckerTest, QueryElevationComponentTest, ShowFileRequestHandlerTest, SearchHandlerTest, FileBasedSpellCheckerTest, SimpleFacetsTest, TestConfig, DocValuesTest, CircularListTest, ShardRoutingTest, SliceStateTest, RegexBoostProcessorTest, PluginInfoTest, TestFieldCollectionResource, TestDocSet, TestSolrIndexConfig, ZkControllerTest, BasicDistributedZkTest, ZkCLITest, BasicFunctionalityTest, CacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, SuggesterFSTTest, SolrInfoMBeanTest, JsonLoaderTest, PeerSyncTest, QueryResultKeyTest, StatelessScriptUpdateProcessorFactoryTest, OverseerTest, MBeansHandlerTest, DocumentBuilderTest, TestDynamicFieldResource, TestPluginEnable, SpatialFilterTest, DirectUpdateHandlerOptimizeTest, ChaosMonkeyNothingIsSafeTest, TestFunctionQuery, PreAnalyzedFieldTest, TestLFUCache, TestCharFilters, TestMaxScoreQueryParser, TestPhraseSuggestions, DistributedQueryElevationComponentTest, TestRealTimeGet, TestRemoteStreaming, ShardSplitTest, CSVRequestHandlerTest, TestSurroundQueryParser, TestQueryUtils, SolrTestCaseJ4Test, TestCoreContainer, RAMDirectoryFactoryTest, TestReversedWildcardFilterFactory, QueryEqualityTest, TestXIncludeConfig, PolyFieldTest, SOLR749Test, UnloadDistributedZkTest, TestJoin, TestSchemaResource, IndexSchemaTest, BadComponentTest, FastVectorHighlighterTest, TestPHPSerializedResponseWriter, TestRandomDVFaceting, TestWriterPerf, SolrCoreTest, SyncSliceTest, RecoveryZkTest]
[junit4:junit4] Completed on J0 in 144.77s, 1 test, 1 failure <<< FAILURES!
[...truncated 440 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:385: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:365: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 2 failures, 13 ignored (7 assumptions)
Total time: 56 minutes 6 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4066 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4066/
1 tests failed.
FAILED: org.apache.solr.cloud.ShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:43697/ui returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:43697/ui returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([F698F222C00ECA43:777E7C3AB751AA7F]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 10046 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ShardSplitTest
[junit4:junit4] 2> 1707640 T5014 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ui/
[junit4:junit4] 2> 1707646 T5014 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./solrtest-ShardSplitTest-1371250583809
[junit4:junit4] 2> 1707648 T5014 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 1707649 T5015 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 1707751 T5014 oasc.ZkTestServer.run start zk server on port:35855
[junit4:junit4] 2> 1707752 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1707758 T5021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bc4572d name:ZooKeeperConnection Watcher:127.0.0.1:35855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1707758 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1707758 T5014 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 1707776 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1707779 T5023 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f76bd44 name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1707779 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1707780 T5014 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 1707783 T5014 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 1707792 T5014 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 1707795 T5014 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 1707798 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1707799 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1707810 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 1707811 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 1707916 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1707917 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1707926 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 1707927 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 1707930 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 1707931 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 1707934 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1707935 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1707938 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1707938 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1707941 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1707942 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1707945 T5014 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1707946 T5014 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1708251 T5014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1708255 T5014 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42036
[junit4:junit4] 2> 1708256 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1708257 T5014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1708257 T5014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118
[junit4:junit4] 2> 1708258 T5014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/solr.xml
[junit4:junit4] 2> 1708258 T5014 oasc.CoreContainer.<init> New CoreContainer 1183385978
[junit4:junit4] 2> 1708259 T5014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/'
[junit4:junit4] 2> 1708260 T5014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/'
[junit4:junit4] 2> 1708395 T5014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1708396 T5014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1708397 T5014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1708397 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1708398 T5014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1708399 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1708399 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1708400 T5014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1708400 T5014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1708401 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1708419 T5014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1708420 T5014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35855/solr
[junit4:junit4] 2> 1708420 T5014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1708422 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1708425 T5034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23e35a7f name:ZooKeeperConnection Watcher:127.0.0.1:35855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1708426 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1708428 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1708441 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1708443 T5036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@706681e8 name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1708443 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1708445 T5014 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 1708455 T5014 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 1708459 T5014 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 1708465 T5014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42036_ui
[junit4:junit4] 2> 1708467 T5014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42036_ui
[junit4:junit4] 2> 1708470 T5014 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 1708486 T5014 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 1708494 T5014 oasc.Overseer.start Overseer (id=89866278260965379-127.0.0.1:42036_ui-n_0000000000) starting
[junit4:junit4] 2> 1708499 T5014 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 1708510 T5038 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 1708511 T5014 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 1708513 T5014 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 1708521 T5014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1708524 T5037 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 1708529 T5039 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1708529 T5039 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1710027 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1710028 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42036_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42036/ui"}
[junit4:junit4] 2> 1710029 T5037 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1710029 T5037 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1710046 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1710532 T5039 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/collection1
[junit4:junit4] 2> 1710532 T5039 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 1710533 T5039 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1710533 T5039 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 1710535 T5039 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/collection1/'
[junit4:junit4] 2> 1710536 T5039 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/collection1/lib/README' to classloader
[junit4:junit4] 2> 1710537 T5039 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1710595 T5039 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 1710663 T5039 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1710766 T5039 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1710774 T5039 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1711389 T5039 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1711393 T5039 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1711395 T5039 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1711401 T5039 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1711424 T5039 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1711425 T5039 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371250584118/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/control/data/
[junit4:junit4] 2> 1711425 T5039 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@84b2f7f
[junit4:junit4] 2> 1711426 T5039 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1711426 T5039 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/control/data
[junit4:junit4] 2> 1711427 T5039 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371250583808/control/data/index/
[junit4:junit4] 2> 1711427 T5039 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371250583808/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1711428 T5039 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/control/data/index
[junit4:junit4] 2> 1711432 T5039 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7482441f lockFactory=org.apache.lucene.store.NativeFSLockFactory@726274f0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1711433 T5039 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1711436 T5039 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1711437 T5039 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1711437 T5039 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1711438 T5039 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1711439 T5039 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1711439 T5039 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1711439 T5039 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1711440 T5039 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1711440 T5039 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1711457 T5039 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1711465 T5039 oass.SolrIndexSearcher.<init> Opening Searcher@2926d8b0 main
[junit4:junit4] 2> 1711465 T5039 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/control/data/tlog
[junit4:junit4] 2> 1711466 T5039 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1711467 T5039 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1711470 T5040 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2926d8b0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1711472 T5039 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1711473 T5039 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42036/ui collection:control_collection shard:shard1
[junit4:junit4] 2> 1711474 T5039 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 1711495 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1711498 T5039 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1711498 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1711499 T5039 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42036/ui/collection1/
[junit4:junit4] 2> 1711499 T5039 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1711499 T5039 oasc.SyncStrategy.syncToMe http://127.0.0.1:42036/ui/collection1/ has no replicas
[junit4:junit4] 2> 1711500 T5039 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42036/ui/collection1/
[junit4:junit4] 2> 1711500 T5039 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 1711550 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1711557 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1711608 T5039 oasc.ZkController.register We are http://127.0.0.1:42036/ui/collection1/ and leader is http://127.0.0.1:42036/ui/collection1/
[junit4:junit4] 2> 1711608 T5039 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42036/ui
[junit4:junit4] 2> 1711608 T5039 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1711608 T5039 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1711609 T5039 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1711610 T5039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1711612 T5014 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1711612 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1711613 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1711631 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 1711633 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1711635 T5043 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@769cf7e1 name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1711635 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1711637 T5014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1711639 T5014 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 1711890 T5014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1711894 T5014 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43697
[junit4:junit4] 2> 1711895 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1711895 T5014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1711896 T5014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800
[junit4:junit4] 2> 1711896 T5014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/solr.xml
[junit4:junit4] 2> 1711897 T5014 oasc.CoreContainer.<init> New CoreContainer 1026292742
[junit4:junit4] 2> 1711898 T5014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/'
[junit4:junit4] 2> 1711898 T5014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/'
[junit4:junit4] 2> 1712042 T5014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1712043 T5014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1712044 T5014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1712044 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1712045 T5014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1712046 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1712046 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1712047 T5014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1712047 T5014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1712048 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1712066 T5014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1712067 T5014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35855/solr
[junit4:junit4] 2> 1712067 T5014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1712068 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1712162 T5054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@372dcd1b name:ZooKeeperConnection Watcher:127.0.0.1:35855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1712163 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1712165 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1712178 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1712180 T5056 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55751633 name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1712181 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1712225 T5014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1713062 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1713063 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42036_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42036/ui"}
[junit4:junit4] 2> 1713083 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1713083 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1713083 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1713229 T5014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43697_ui
[junit4:junit4] 2> 1713231 T5014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43697_ui
[junit4:junit4] 2> 1713233 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1713234 T5036 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1713234 T5056 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1713234 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1713234 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1713235 T5043 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1713249 T5057 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1713250 T5057 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1714588 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1714589 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43697_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43697/ui"}
[junit4:junit4] 2> 1714589 T5037 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 1714589 T5037 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1714606 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1714606 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1714606 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1715253 T5057 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/collection1
[junit4:junit4] 2> 1715253 T5057 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1715254 T5057 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1715254 T5057 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1715256 T5057 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/collection1/'
[junit4:junit4] 2> 1715257 T5057 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1715258 T5057 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/collection1/lib/README' to classloader
[junit4:junit4] 2> 1715317 T5057 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 1715408 T5057 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1715511 T5057 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1715517 T5057 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1716098 T5057 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1716101 T5057 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1716103 T5057 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1716107 T5057 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1716127 T5057 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1716128 T5057 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371250587800/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty1/
[junit4:junit4] 2> 1716128 T5057 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@84b2f7f
[junit4:junit4] 2> 1716128 T5057 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1716129 T5057 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty1
[junit4:junit4] 2> 1716129 T5057 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty1/index/
[junit4:junit4] 2> 1716130 T5057 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1716130 T5057 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty1/index
[junit4:junit4] 2> 1716133 T5057 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368d0be lockFactory=org.apache.lucene.store.NativeFSLockFactory@66950fac),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1716133 T5057 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1716135 T5057 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1716136 T5057 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1716136 T5057 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1716137 T5057 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1716138 T5057 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1716138 T5057 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1716138 T5057 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1716138 T5057 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1716139 T5057 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1716150 T5057 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1716155 T5057 oass.SolrIndexSearcher.<init> Opening Searcher@7c9accc9 main
[junit4:junit4] 2> 1716155 T5057 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty1/tlog
[junit4:junit4] 2> 1716156 T5057 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1716156 T5057 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1716160 T5058 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c9accc9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1716161 T5057 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1716162 T5057 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43697/ui collection:collection1 shard:shard1
[junit4:junit4] 2> 1716162 T5057 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 1716181 T5057 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1716184 T5057 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1716184 T5057 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1716184 T5057 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43697/ui/collection1/
[junit4:junit4] 2> 1716185 T5057 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1716185 T5057 oasc.SyncStrategy.syncToMe http://127.0.0.1:43697/ui/collection1/ has no replicas
[junit4:junit4] 2> 1716185 T5057 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43697/ui/collection1/
[junit4:junit4] 2> 1716186 T5057 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1717611 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1717643 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1717643 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1717643 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1717672 T5057 oasc.ZkController.register We are http://127.0.0.1:43697/ui/collection1/ and leader is http://127.0.0.1:43697/ui/collection1/
[junit4:junit4] 2> 1717672 T5057 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43697/ui
[junit4:junit4] 2> 1717672 T5057 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1717673 T5057 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1717673 T5057 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1717674 T5057 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1717676 T5014 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1717676 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1717677 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1717945 T5014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1717948 T5014 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:33218
[junit4:junit4] 2> 1717949 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1717950 T5014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1717950 T5014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856
[junit4:junit4] 2> 1717951 T5014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/solr.xml
[junit4:junit4] 2> 1717952 T5014 oasc.CoreContainer.<init> New CoreContainer 2078832550
[junit4:junit4] 2> 1717952 T5014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/'
[junit4:junit4] 2> 1717953 T5014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/'
[junit4:junit4] 2> 1718088 T5014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1718089 T5014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1718089 T5014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1718090 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1718090 T5014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1718091 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1718091 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1718092 T5014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1718093 T5014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1718093 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1718111 T5014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1718112 T5014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35855/solr
[junit4:junit4] 2> 1718112 T5014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1718113 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1718117 T5070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@795b59a4 name:ZooKeeperConnection Watcher:127.0.0.1:35855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1718118 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1718135 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1718147 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1718149 T5072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1218fe11 name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1718150 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1718161 T5014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1719158 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1719159 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43697_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43697/ui"}
[junit4:junit4] 2> 1719176 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1719176 T5014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33218_ui
[junit4:junit4] 2> 1719176 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1719176 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1719176 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1719184 T5014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33218_ui
[junit4:junit4] 2> 1719187 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1719187 T5056 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1719187 T5072 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1719187 T5036 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1719188 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1719188 T5043 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1719188 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1719188 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1719196 T5073 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1719197 T5073 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1720687 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1720688 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33218_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33218/ui"}
[junit4:junit4] 2> 1720689 T5037 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 1720689 T5037 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 1720737 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1720737 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1720737 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1720737 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1721200 T5073 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/collection1
[junit4:junit4] 2> 1721200 T5073 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1721201 T5073 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1721201 T5073 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1721203 T5073 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/collection1/'
[junit4:junit4] 2> 1721204 T5073 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/collection1/lib/README' to classloader
[junit4:junit4] 2> 1721205 T5073 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1721261 T5073 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 1721327 T5073 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1721429 T5073 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1721435 T5073 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1722056 T5073 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1722060 T5073 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1722063 T5073 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1722068 T5073 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1722091 T5073 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1722092 T5073 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371250593856/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty2/
[junit4:junit4] 2> 1722092 T5073 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@84b2f7f
[junit4:junit4] 2> 1722093 T5073 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1722095 T5073 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty2
[junit4:junit4] 2> 1722095 T5073 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty2/index/
[junit4:junit4] 2> 1722095 T5073 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1722096 T5073 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty2/index
[junit4:junit4] 2> 1722100 T5073 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f5ebaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ec5dfa),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1722100 T5073 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1722104 T5073 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1722105 T5073 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1722106 T5073 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1722106 T5073 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1722107 T5073 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1722107 T5073 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1722108 T5073 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1722108 T5073 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1722109 T5073 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1722125 T5073 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1722133 T5073 oass.SolrIndexSearcher.<init> Opening Searcher@7e848016 main
[junit4:junit4] 2> 1722134 T5073 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty2/tlog
[junit4:junit4] 2> 1722135 T5073 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1722135 T5073 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1722140 T5074 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e848016 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1722142 T5073 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1722143 T5073 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33218/ui collection:collection1 shard:shard2
[junit4:junit4] 2> 1722144 T5073 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 1722253 T5073 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1722316 T5073 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1722317 T5073 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1722317 T5073 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33218/ui/collection1/
[junit4:junit4] 2> 1722317 T5073 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1722318 T5073 oasc.SyncStrategy.syncToMe http://127.0.0.1:33218/ui/collection1/ has no replicas
[junit4:junit4] 2> 1722318 T5073 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33218/ui/collection1/
[junit4:junit4] 2> 1722318 T5073 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 1723756 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1723784 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1723784 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1723784 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1723784 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1723798 T5073 oasc.ZkController.register We are http://127.0.0.1:33218/ui/collection1/ and leader is http://127.0.0.1:33218/ui/collection1/
[junit4:junit4] 2> 1723798 T5073 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33218/ui
[junit4:junit4] 2> 1723798 T5073 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1723799 T5073 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1723799 T5073 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1723800 T5073 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1723802 T5014 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1723802 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1723803 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1724071 T5014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1724075 T5014 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:21316
[junit4:junit4] 2> 1724076 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1724077 T5014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1724077 T5014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982
[junit4:junit4] 2> 1724078 T5014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/solr.xml
[junit4:junit4] 2> 1724078 T5014 oasc.CoreContainer.<init> New CoreContainer 409495341
[junit4:junit4] 2> 1724079 T5014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/'
[junit4:junit4] 2> 1724080 T5014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/'
[junit4:junit4] 2> 1724212 T5014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1724213 T5014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1724213 T5014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1724214 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1724214 T5014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1724215 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1724216 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1724216 T5014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1724217 T5014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1724217 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1724233 T5014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1724234 T5014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35855/solr
[junit4:junit4] 2> 1724234 T5014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1724236 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1724239 T5086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57fd30e2 name:ZooKeeperConnection Watcher:127.0.0.1:35855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1724240 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1724253 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1724264 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1724266 T5088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a85ab50 name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1724267 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1724278 T5014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1725281 T5014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:21316_ui
[junit4:junit4] 2> 1725283 T5014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:21316_ui
[junit4:junit4] 2> 1725285 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1725286 T5088 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1725286 T5072 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1725286 T5036 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1725286 T5056 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1725287 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725287 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725287 T5043 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1725287 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725299 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1725300 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33218_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33218/ui"}
[junit4:junit4] 2> 1725304 T5089 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1725304 T5089 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1725309 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:21316_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:21316/ui"}
[junit4:junit4] 2> 1725309 T5037 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 1725310 T5037 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1725319 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725319 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725319 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725319 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1725319 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1726310 T5089 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/collection1
[junit4:junit4] 2> 1726310 T5089 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1726311 T5089 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1726311 T5089 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1726313 T5089 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/collection1/'
[junit4:junit4] 2> 1726314 T5089 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/collection1/lib/README' to classloader
[junit4:junit4] 2> 1726315 T5089 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1726370 T5089 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 1726433 T5089 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1726536 T5089 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1726542 T5089 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1727148 T5089 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1727152 T5089 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1727154 T5089 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1727160 T5089 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1727183 T5089 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1727183 T5089 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371250599982/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/
[junit4:junit4] 2> 1727183 T5089 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@84b2f7f
[junit4:junit4] 2> 1727184 T5089 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1727185 T5089 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3
[junit4:junit4] 2> 1727185 T5089 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index/
[junit4:junit4] 2> 1727185 T5089 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1727186 T5089 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index
[junit4:junit4] 2> 1727189 T5089 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@79358abd lockFactory=org.apache.lucene.store.NativeFSLockFactory@204aac7d),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1727190 T5089 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1727193 T5089 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1727194 T5089 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1727194 T5089 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1727195 T5089 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1727196 T5089 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1727196 T5089 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1727196 T5089 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1727197 T5089 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1727198 T5089 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1727213 T5089 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1727220 T5089 oass.SolrIndexSearcher.<init> Opening Searcher@6f118085 main
[junit4:junit4] 2> 1727221 T5089 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/tlog
[junit4:junit4] 2> 1727222 T5089 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1727222 T5089 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1727227 T5090 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f118085 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1727229 T5089 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1727229 T5089 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:21316/ui collection:collection1 shard:shard1
[junit4:junit4] 2> 1727234 T5089 oasc.ZkController.register We are http://127.0.0.1:21316/ui/collection1/ and leader is http://127.0.0.1:43697/ui/collection1/
[junit4:junit4] 2> 1727234 T5089 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:21316/ui
[junit4:junit4] 2> 1727234 T5089 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 1727235 T5089 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1097 name=collection1 org.apache.solr.core.SolrCore@921733 url=http://127.0.0.1:21316/ui/collection1 node=127.0.0.1:21316_ui C1097_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:21316_ui, base_url=http://127.0.0.1:21316/ui}
[junit4:junit4] 2> 1727235 T5091 C1097 P21316 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 1727235 T5089 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1727236 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 1727237 T5091 C1097 P21316 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 1727237 T5091 C1097 P21316 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1727237 T5014 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1727238 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1727238 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1727297 T5091 C1097 P21316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1727309 T5046 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 1727488 T5014 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1727492 T5014 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:10474
[junit4:junit4] 2> 1727493 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1727494 T5014 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1727494 T5014 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414
[junit4:junit4] 2> 1727495 T5014 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/solr.xml
[junit4:junit4] 2> 1727495 T5014 oasc.CoreContainer.<init> New CoreContainer 1464550547
[junit4:junit4] 2> 1727496 T5014 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/'
[junit4:junit4] 2> 1727496 T5014 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/'
[junit4:junit4] 2> 1727620 T5014 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1727621 T5014 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1727622 T5014 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1727622 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1727623 T5014 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1727623 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1727624 T5014 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1727624 T5014 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1727625 T5014 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1727626 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1727640 T5014 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1727641 T5014 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35855/solr
[junit4:junit4] 2> 1727642 T5014 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1727643 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1727649 T5103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26b4de8c name:ZooKeeperConnection Watcher:127.0.0.1:35855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1727650 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1727665 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1727683 T5014 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1727685 T5105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a67558c name:ZooKeeperConnection Watcher:127.0.0.1:35855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1727685 T5014 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1727697 T5014 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1728332 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1728333 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:21316_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:21316/ui"}
[junit4:junit4] 2> 1728375 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728375 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728375 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728375 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728375 T5105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728375 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728701 T5014 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:10474_ui
[junit4:junit4] 2> 1728703 T5014 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:10474_ui
[junit4:junit4] 2> 1728705 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728705 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728706 T5056 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1728706 T5072 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1728706 T5036 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1728705 T5105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1728706 T5043 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1728706 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1728706 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1728706 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1728720 T5088 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1728721 T5105 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1728725 T5106 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1728726 T5106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1729312 T5046 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 1729312 T5046 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:21316_ui&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 1729879 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1729880 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10474_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10474/ui"}
[junit4:junit4] 2> 1729881 T5037 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 1729881 T5037 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 1729898 T5105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1729898 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1729898 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1729898 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1729898 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1729898 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1730729 T5106 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/collection1
[junit4:junit4] 2> 1730729 T5106 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1730730 T5106 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1730730 T5106 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1730732 T5106 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/collection1/'
[junit4:junit4] 2> 1730733 T5106 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/collection1/lib/README' to classloader
[junit4:junit4] 2> 1730734 T5106 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1730789 T5106 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 1730851 T5106 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1730954 T5106 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1730961 T5106 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> C1097_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:21316_ui, base_url=http://127.0.0.1:21316/ui}
[junit4:junit4] 2> 1731314 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43697/ui/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 1731314 T5091 C1097 P21316 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:21316/ui START replicas=[http://127.0.0.1:43697/ui/collection1/] nUpdates=100
[junit4:junit4] 2> 1731315 T5091 C1097 P21316 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 1731315 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 1731315 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 1731315 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 1731315 T5091 C1097 P21316 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 1731316 T5091 C1097 P21316 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43697/ui/collection1/. core=collection1
[junit4:junit4] 2> 1731316 T5091 C1097 P21316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1098 name=collection1 org.apache.solr.core.SolrCore@1b784cfa url=http://127.0.0.1:43697/ui/collection1 node=127.0.0.1:43697_ui C1098_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43697_ui, base_url=http://127.0.0.1:43697/ui, leader=true}
[junit4:junit4] 2> 1731324 T5047 C1098 P43697 oasc.SolrCore.execute [collection1] webapp=/ui path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 1731326 T5048 C1098 P43697 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1731329 T5048 C1098 P43697 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368d0be lockFactory=org.apache.lucene.store.NativeFSLockFactory@66950fac),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1731329 T5048 C1098 P43697 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1731330 T5048 C1098 P43697 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368d0be lockFactory=org.apache.lucene.store.NativeFSLockFactory@66950fac),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368d0be lockFactory=org.apache.lucene.store.NativeFSLockFactory@66950fac),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1731330 T5048 C1098 P43697 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1731331 T5048 C1098 P43697 oass.SolrIndexSearcher.<init> Opening Searcher@295c9936 realtime
[junit4:junit4] 2> 1731331 T5048 C1098 P43697 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1731331 T5048 C1098 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 1731332 T5091 C1097 P21316 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1731332 T5091 C1097 P21316 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 1731334 T5049 C1098 P43697 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1731334 T5049 C1098 P43697 oasc.SolrCore.execute [collection1] webapp=/ui path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 1731335 T5091 C1097 P21316 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 1731335 T5091 C1097 P21316 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 1731335 T5091 C1097 P21316 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 1731337 T5049 C1098 P43697 oasc.SolrCore.execute [collection1] webapp=/ui path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 1731337 T5091 C1097 P21316 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 1731339 T5091 C1097 P21316 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index.20130615105647499
[junit4:junit4] 2> 1731339 T5091 C1097 P21316 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@274b6b21 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d2fee35) fullCopy=false
[junit4:junit4] 2> 1731342 T5049 C1098 P43697 oasc.SolrCore.execute [collection1] webapp=/ui path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1731343 T5091 C1097 P21316 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 1731344 T5091 C1097 P21316 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 1731344 T5091 C1097 P21316 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 1731346 T5091 C1097 P21316 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@79358abd lockFactory=org.apache.lucene.store.NativeFSLockFactory@204aac7d),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@79358abd lockFactory=org.apache.lucene.store.NativeFSLockFactory@204aac7d),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1731346 T5091 C1097 P21316 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1731347 T5091 C1097 P21316 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 1731347 T5091 C1097 P21316 oass.SolrIndexSearcher.<init> Opening Searcher@7f04396 main
[junit4:junit4] 2> 1731348 T5090 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f04396 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1731348 T5091 C1097 P21316 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index.20130615105647499 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index.20130615105647499;done=true>>]
[junit4:junit4] 2> 1731349 T5091 C1097 P21316 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index.20130615105647499
[junit4:junit4] 2> 1731349 T5091 C1097 P21316 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index.20130615105647499
[junit4:junit4] 2> 1731349 T5091 C1097 P21316 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 1731349 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 1731349 T5091 C1097 P21316 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1731350 T5091 C1097 P21316 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1731351 T5091 C1097 P21316 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 1731403 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1731404 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:21316_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:21316/ui"}
[junit4:junit4] 2> 1731416 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1731416 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1731416 T5105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1731416 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1731416 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1731416 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1731544 T5106 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1731548 T5106 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1731550 T5106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1731554 T5106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1731598 T5106 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1731599 T5106 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371250603414/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/
[junit4:junit4] 2> 1731599 T5106 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@84b2f7f
[junit4:junit4] 2> 1731600 T5106 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1731601 T5106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4
[junit4:junit4] 2> 1731601 T5106 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index/
[junit4:junit4] 2> 1731601 T5106 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1731602 T5106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index
[junit4:junit4] 2> 1731606 T5106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@51f194c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bc18fc1),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1731606 T5106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1731610 T5106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1731610 T5106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1731611 T5106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1731612 T5106 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1731613 T5106 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1731613 T5106 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1731614 T5106 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1731615 T5106 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1731615 T5106 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1731632 T5106 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1731640 T5106 oass.SolrIndexSearcher.<init> Opening Searcher@2aeea191 main
[junit4:junit4] 2> 1731640 T5106 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/tlog
[junit4:junit4] 2> 1731641 T5106 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1731642 T5106 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1731646 T5109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2aeea191 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1731648 T5106 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1731649 T5106 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:10474/ui collection:collection1 shard:shard2
[junit4:junit4] 2> 1731651 T5106 oasc.ZkController.register We are http://127.0.0.1:10474/ui/collection1/ and leader is http://127.0.0.1:33218/ui/collection1/
[junit4:junit4] 2> 1731652 T5106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:10474/ui
[junit4:junit4] 2> 1731652 T5106 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 1731652 T5106 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1099 name=collection1 org.apache.solr.core.SolrCore@38cdc525 url=http://127.0.0.1:10474/ui/collection1 node=127.0.0.1:10474_ui C1099_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:10474_ui, base_url=http://127.0.0.1:10474/ui}
[junit4:junit4] 2> 1731653 T5110 C1099 P10474 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 1731653 T5106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1731653 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 1731654 T5110 C1099 P10474 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 1731654 T5110 C1099 P10474 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1731654 T5014 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1731655 T5014 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1731655 T5110 C1099 P10474 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1731655 T5014 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1731669 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1731670 T5062 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 1731670 T5014 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 1731671 T5014 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 1731672 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1732673 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1732921 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1732922 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10474_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10474/ui"}
[junit4:junit4] 2> 1732959 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1732959 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1732959 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1732959 T5105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1732959 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1732959 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1733673 T5062 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 1733673 T5062 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:10474_ui&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 1733676 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1734678 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C1099_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:10474_ui, base_url=http://127.0.0.1:10474/ui}
[junit4:junit4] 2> 1735675 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33218/ui/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 1735675 T5110 C1099 P10474 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:10474/ui START replicas=[http://127.0.0.1:33218/ui/collection1/] nUpdates=100
[junit4:junit4] 2> 1735676 T5110 C1099 P10474 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 1735676 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 1735677 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 1735677 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 1735677 T5110 C1099 P10474 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 1735678 T5110 C1099 P10474 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33218/ui/collection1/. core=collection1
[junit4:junit4] 2> 1735678 T5110 C1099 P10474 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1735681 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1100 name=collection1 org.apache.solr.core.SolrCore@4d11a07d url=http://127.0.0.1:33218/ui/collection1 node=127.0.0.1:33218_ui C1100_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:33218_ui, base_url=http://127.0.0.1:33218/ui, leader=true}
[junit4:junit4] 2> 1735688 T5063 C1100 P33218 oasc.SolrCore.execute [collection1] webapp=/ui path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 1735692 T5064 C1100 P33218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1735695 T5064 C1100 P33218 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f5ebaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ec5dfa),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1735696 T5064 C1100 P33218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1735696 T5064 C1100 P33218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f5ebaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ec5dfa),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f5ebaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ec5dfa),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1735697 T5064 C1100 P33218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1735698 T5064 C1100 P33218 oass.SolrIndexSearcher.<init> Opening Searcher@657ede29 realtime
[junit4:junit4] 2> 1735698 T5064 C1100 P33218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1735699 T5064 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 1735700 T5110 C1099 P10474 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1735700 T5110 C1099 P10474 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 1735702 T5065 C1100 P33218 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1735703 T5065 C1100 P33218 oasc.SolrCore.execute [collection1] webapp=/ui path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 1735703 T5110 C1099 P10474 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 1735704 T5110 C1099 P10474 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 1735704 T5110 C1099 P10474 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 1735706 T5065 C1100 P33218 oasc.SolrCore.execute [collection1] webapp=/ui path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 1735707 T5110 C1099 P10474 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 1735709 T5110 C1099 P10474 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index.20130615105651869
[junit4:junit4] 2> 1735709 T5110 C1099 P10474 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f43b7d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2eaebc31) fullCopy=false
[junit4:junit4] 2> 1735712 T5065 C1100 P33218 oasc.SolrCore.execute [collection1] webapp=/ui path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1735714 T5110 C1099 P10474 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 1735716 T5110 C1099 P10474 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 1735716 T5110 C1099 P10474 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 1735718 T5110 C1099 P10474 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@51f194c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bc18fc1),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@51f194c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bc18fc1),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1735718 T5110 C1099 P10474 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1735719 T5110 C1099 P10474 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 1735719 T5110 C1099 P10474 oass.SolrIndexSearcher.<init> Opening Searcher@dac1228 main
[junit4:junit4] 2> 1735720 T5109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@dac1228 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1735721 T5110 C1099 P10474 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index.20130615105651869 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index.20130615105651869;done=true>>]
[junit4:junit4] 2> 1735721 T5110 C1099 P10474 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index.20130615105651869
[junit4:junit4] 2> 1735722 T5110 C1099 P10474 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty4/index.20130615105651869
[junit4:junit4] 2> 1735722 T5110 C1099 P10474 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 1735722 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 1735723 T5110 C1099 P10474 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1735723 T5110 C1099 P10474 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1735725 T5110 C1099 P10474 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 1735965 T5037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1735966 T5037 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:10474_ui",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:10474/ui"}
[junit4:junit4] 2> 1735981 T5105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1735981 T5072 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1735981 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1735981 T5043 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1735981 T5036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1735981 T5056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1736684 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1736686 T5014 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C1101 name=collection1 org.apache.solr.core.SolrCore@62b10dc url=http://127.0.0.1:42036/ui/collection1 node=127.0.0.1:42036_ui C1101_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:42036_ui, base_url=http://127.0.0.1:42036/ui, leader=true}
[junit4:junit4] 2> 1736695 T5026 C1101 P42036 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1736698 T5026 C1101 P42036 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7482441f lockFactory=org.apache.lucene.store.NativeFSLockFactory@726274f0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1736700 T5026 C1101 P42036 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1736701 T5026 C1101 P42036 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7482441f lockFactory=org.apache.lucene.store.NativeFSLockFactory@726274f0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7482441f lockFactory=org.apache.lucene.store.NativeFSLockFactory@726274f0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1736701 T5026 C1101 P42036 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1736702 T5026 C1101 P42036 oass.SolrIndexSearcher.<init> Opening Searcher@47452de9 main
[junit4:junit4] 2> 1736702 T5026 C1101 P42036 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1736703 T5040 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47452de9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1736705 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 1736717 T5066 C1100 P33218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1736718 T5066 C1100 P33218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f5ebaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ec5dfa),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f5ebaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@27ec5dfa),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1736718 T5066 C1100 P33218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1736719 T5066 C1100 P33218 oass.SolrIndexSearcher.<init> Opening Searcher@2d0fc05b main
[junit4:junit4] 2> 1736719 T5066 C1100 P33218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1736720 T5074 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d0fc05b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1736721 T5066 C1100 P33218 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43697/ui/collection1/, StdNode: http://127.0.0.1:21316/ui/collection1/, StdNode: http://127.0.0.1:10474/ui/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C1102 name=collection1 org.apache.solr.core.SolrCore@1b784cfa url=http://127.0.0.1:43697/ui/collection1 node=127.0.0.1:43697_ui C1102_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43697_ui, base_url=http://127.0.0.1:43697/ui, leader=true}
[junit4:junit4] 2> 1736724 T5050 C1102 P43697 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1736726 T5050 C1102 P43697 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368d0be lockFactory=org.apache.lucene.store.NativeFSLockFactory@66950fac),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1368d0be lockFactory=org.apache.lucene.store.NativeFSLockFactory@66950fac),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1736726 T5050 C1102 P43697 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1736727 T5050 C1102 P43697 oass.SolrIndexSearcher.<init> Opening Searcher@42cddd07 main
[junit4:junit4] 2> 1736727 T5050 C1102 P43697 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1736728 T5058 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@42cddd07 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1736729 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> C1099_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:10474_ui, base_url=http://127.0.0.1:10474/ui}
[junit4:junit4] 2> 1736732 T5095 C1099 P10474 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C1103 name=collection1 org.apache.solr.core.SolrCore@921733 url=http://127.0.0.1:21316/ui/collection1 node=127.0.0.1:21316_ui C1103_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:21316_ui, base_url=http://127.0.0.1:21316/ui}
[junit4:junit4] 2> 1736732 T5078 C1103 P21316 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1736734 T5095 C1099 P10474 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@51f194c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bc18fc1),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@51f194c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bc18fc1),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1736734 T5078 C1103 P21316 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@79358abd lockFactory=org.apache.lucene.store.NativeFSLockFactory@204aac7d),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@79358abd lockFactory=org.apache.lucene.store.NativeFSLockFactory@204aac7d),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1736734 T5095 C1099 P10474 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1736735 T5078 C1103 P21316 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1736736 T5095 C1099 P10474 oass.SolrIndexSearcher.<init> Opening Searcher@43a365f7 main
[junit4:junit4] 2> 1736736 T5078 C1103 P21316 oass.SolrIndexSearcher.<init> Opening Searcher@6b2d6c1e main
[junit4:junit4] 2> 1736737 T5095 C1099 P10474 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1736738 T5109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43a365f7 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1736738 T5090 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b2d6c1e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1736737 T5078 C1103 P21316 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1736738 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 1736739 T5078 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 1736740 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4] 2> 1736741 T5014 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1736744 T5051 C1102 P43697 oasc.SolrCore.execute [collection1] webapp=/ui path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 1736746 T5079 C1103 P21316 oasc.SolrCore.execute [collection1] webapp=/ui path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 1736748 T5067 C1100 P33218 oasc.SolrCore.execute [collection1] webapp=/ui path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 1736751 T5096 C1099 P10474 oasc.SolrCore.execute [collection1] webapp=/ui path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 1738757 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437860484787208192)} 0 2
[junit4:junit4] 2> 1738767 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={update.distrib=FROMLEADER&_version_=-1437860484791402496&update.from=http://127.0.0.1:43697/ui/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437860484791402496)} 0 2
[junit4:junit4] 2> 1738770 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={update.distrib=FROMLEADER&_version_=-1437860484796645376&update.from=http://127.0.0.1:33218/ui/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437860484796645376)} 0 1
[junit4:junit4] 2> 1738771 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1437860484796645376)} 0 6
[junit4:junit4] 2> 1738772 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437860484791402496)} 0 12
[junit4:junit4] 2> 1738776 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[a!0 (1437860484807131136)]} 0 1
[junit4:junit4] 2> 1738783 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!0 (1437860484811325440)]} 0 1
[junit4:junit4] 2> 1738783 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[a!0 (1437860484811325440)]} 0 4
[junit4:junit4] 2> 1738786 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[b!1 (1437860484818665472)]} 0 0
[junit4:junit4] 2> 1738795 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!1 (1437860484823908352)]} 0 1
[junit4:junit4] 2> 1738795 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[b!1 (1437860484823908352)]} 0 4
[junit4:junit4] 2> 1738796 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[b!1]} 0 7
[junit4:junit4] 2> 1738799 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[c!2 (1437860484832296960)]} 0 0
[junit4:junit4] 2> 1738805 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!2 (1437860484835442688)]} 0 0
[junit4:junit4] 2> 1738806 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[c!2 (1437860484835442688)]} 0 4
[junit4:junit4] 2> 1738809 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[d!3 (1437860484842782720)]} 0 0
[junit4:junit4] 2> 1738818 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!3 (1437860484849074176)]} 0 0
[junit4:junit4] 2> 1738819 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[d!3 (1437860484849074176)]} 0 4
[junit4:junit4] 2> 1738820 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[d!3]} 0 8
[junit4:junit4] 2> 1738823 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[e!4 (1437860484857462784)]} 0 0
[junit4:junit4] 2> 1738830 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!4 (1437860484860608512)]} 0 1
[junit4:junit4] 2> 1738830 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[e!4 (1437860484860608512)]} 0 4
[junit4:junit4] 2> 1738833 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[f!5 (1437860484867948544)]} 0 0
[junit4:junit4] 2> 1738841 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!5 (1437860484873191424)]} 0 0
[junit4:junit4] 2> 1738842 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!5 (1437860484873191424)]} 0 4
[junit4:junit4] 2> 1738843 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[f!5]} 0 7
[junit4:junit4] 2> 1738846 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[g!6 (1437860484881580032)]} 0 0
[junit4:junit4] 2> 1738854 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!6 (1437860484886822912)]} 0 0
[junit4:junit4] 2> 1738855 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!6 (1437860484886822912)]} 0 4
[junit4:junit4] 2> 1738856 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[g!6]} 0 7
[junit4:junit4] 2> 1738859 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[h!7 (1437860484895211520)]} 0 0
[junit4:junit4] 2> 1738865 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!7 (1437860484898357248)]} 0 0
[junit4:junit4] 2> 1738866 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[h!7 (1437860484898357248)]} 0 4
[junit4:junit4] 2> 1738869 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[i!8 (1437860484905697280)]} 0 0
[junit4:junit4] 2> 1738877 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!8 (1437860484910940160)]} 0 0
[junit4:junit4] 2> 1738878 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!8 (1437860484910940160)]} 0 4
[junit4:junit4] 2> 1738879 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[i!8]} 0 7
[junit4:junit4] 2> 1738883 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[j!9 (1437860484919328768)]} 0 1
[junit4:junit4] 2> 1738891 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!9 (1437860484925620224)]} 0 0
[junit4:junit4] 2> 1738892 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!9 (1437860484925620224)]} 0 4
[junit4:junit4] 2> 1738893 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[j!9]} 0 7
[junit4:junit4] 2> 1738896 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[k!10 (1437860484934008832)]} 0 0
[junit4:junit4] 2> 1738902 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!10 (1437860484937154560)]} 0 0
[junit4:junit4] 2> 1738903 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[k!10 (1437860484937154560)]} 0 4
[junit4:junit4] 2> 1738907 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[l!11 (1437860484944494592)]} 0 1
[junit4:junit4] 2> 1738915 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!11 (1437860484949737472)]} 0 0
[junit4:junit4] 2> 1738915 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[l!11 (1437860484949737472)]} 0 4
[junit4:junit4] 2> 1738916 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[l!11]} 0 7
[junit4:junit4] 2> 1738919 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[m!12 (1437860484958126080)]} 0 0
[junit4:junit4] 2> 1738925 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!12 (1437860484961271808)]} 0 0
[junit4:junit4] 2> 1738926 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[m!12 (1437860484961271808)]} 0 4
[junit4:junit4] 2> 1738929 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[n!13 (1437860484968611840)]} 0 0
[junit4:junit4] 2> 1738938 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!13 (1437860484973854720)]} 0 1
[junit4:junit4] 2> 1738938 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[n!13 (1437860484973854720)]} 0 4
[junit4:junit4] 2> 1738939 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[n!13]} 0 7
[junit4:junit4] 2> 1738942 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[o!14 (1437860484982243328)]} 0 0
[junit4:junit4] 2> 1738950 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!14 (1437860484987486208)]} 0 0
[junit4:junit4] 2> 1738951 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[o!14 (1437860484987486208)]} 0 4
[junit4:junit4] 2> 1738952 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[o!14]} 0 7
[junit4:junit4] 2> 1738955 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[p!15 (1437860484995874816)]} 0 0
[junit4:junit4] 2> 1738962 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!15 (1437860484999020544)]} 0 1
[junit4:junit4] 2> 1738962 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[p!15 (1437860484999020544)]} 0 4
[junit4:junit4] 2> 1738965 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[q!16 (1437860485006360576)]} 0 0
[junit4:junit4] 2> 1738974 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!16 (1437860485012652032)]} 0 0
[junit4:junit4] 2> 1738975 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[q!16 (1437860485012652032)]} 0 4
[junit4:junit4] 2> 1738976 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[q!16]} 0 7
[junit4:junit4] 2> 1738979 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[r!17 (1437860485021040640)]} 0 0
[junit4:junit4] 2> 1738987 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!17 (1437860485026283520)]} 0 0
[junit4:junit4] 2> 1738988 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[r!17 (1437860485026283520)]} 0 4
[junit4:junit4] 2> 1738989 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[r!17]} 0 7
[junit4:junit4] 2> 1738992 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[s!18 (1437860485034672128)]} 0 0
[junit4:junit4] 2> 1738998 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!18 (1437860485037817856)]} 0 0
[junit4:junit4] 2> 1738999 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[s!18 (1437860485037817856)]} 0 4
[junit4:junit4] 2> 1739002 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[t!19 (1437860485045157888)]} 0 0
[junit4:junit4] 2> 1739008 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!19 (1437860485048303616)]} 0 0
[junit4:junit4] 2> 1739009 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[t!19 (1437860485048303616)]} 0 4
[junit4:junit4] 2> 1739012 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[u!20 (1437860485055643648)]} 0 0
[junit4:junit4] 2> 1739020 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!20 (1437860485060886528)]} 0 0
[junit4:junit4] 2> 1739021 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[u!20 (1437860485060886528)]} 0 4
[junit4:junit4] 2> 1739022 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[u!20]} 0 7
[junit4:junit4] 2> 1739025 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[v!21 (1437860485069275136)]} 0 0
[junit4:junit4] 2> 1739031 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!21 (1437860485072420864)]} 0 0
[junit4:junit4] 2> 1739032 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[v!21 (1437860485072420864)]} 0 4
[junit4:junit4] 2> 1739035 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[w!22 (1437860485079760896)]} 0 0
[junit4:junit4] 2> 1739043 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!22 (1437860485085003776)]} 0 0
[junit4:junit4] 2> 1739044 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[w!22 (1437860485085003776)]} 0 4
[junit4:junit4] 2> 1739045 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[w!22]} 0 7
[junit4:junit4] 2> 1739048 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[x!23 (1437860485093392384)]} 0 0
[junit4:junit4] 2> 1739055 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!23 (1437860485096538112)]} 0 1
[junit4:junit4] 2> 1739056 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[x!23 (1437860485096538112)]} 0 5
[junit4:junit4] 2> 1739058 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[y!24 (1437860485103878144)]} 0 0
[junit4:junit4] 2> 1739067 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!24 (1437860485109121024)]} 0 1
[junit4:junit4] 2> 1739067 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[y!24 (1437860485109121024)]} 0 4
[junit4:junit4] 2> 1739068 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[y!24]} 0 7
[junit4:junit4] 2> 1739072 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[z!25 (1437860485118558208)]} 0 0
[junit4:junit4] 2> 1739081 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!25 (1437860485123801088)]} 0 1
[junit4:junit4] 2> 1739081 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[z!25 (1437860485123801088)]} 0 4
[junit4:junit4] 2> 1739082 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[z!25]} 0 7
[junit4:junit4] 2> 1739085 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[a!26 (1437860485132189696)]} 0 0
[junit4:junit4] 2> 1739091 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!26 (1437860485135335424)]} 0 0
[junit4:junit4] 2> 1739092 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[a!26 (1437860485135335424)]} 0 4
[junit4:junit4] 2> 1739095 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[b!27 (1437860485142675456)]} 0 0
[junit4:junit4] 2> 1739102 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!27 (1437860485145821184)]} 0 1
[junit4:junit4] 2> 1739102 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[b!27 (1437860485145821184)]} 0 4
[junit4:junit4] 2> 1739105 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[c!28 (1437860485153161216)]} 0 0
[junit4:junit4] 2> 1739111 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!28 (1437860485156306944)]} 0 0
[junit4:junit4] 2> 1739112 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[c!28 (1437860485156306944)]} 0 4
[junit4:junit4] 2> 1739115 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[d!29 (1437860485163646976)]} 0 0
[junit4:junit4] 2> 1739123 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!29 (1437860485168889856)]} 0 0
[junit4:junit4] 2> 1739124 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[d!29 (1437860485168889856)]} 0 4
[junit4:junit4] 2> 1739125 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[d!29]} 0 7
[junit4:junit4] 2> 1739128 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[e!30 (1437860485177278464)]} 0 0
[junit4:junit4] 2> 1739134 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!30 (1437860485180424192)]} 0 0
[junit4:junit4] 2> 1739135 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[e!30 (1437860485180424192)]} 0 4
[junit4:junit4] 2> 1739138 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[f!31 (1437860485187764224)]} 0 0
[junit4:junit4] 2> 1739145 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!31 (1437860485190909952)]} 0 1
[junit4:junit4] 2> 1739145 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[f!31 (1437860485190909952)]} 0 4
[junit4:junit4] 2> 1739148 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[g!32 (1437860485198249984)]} 0 0
[junit4:junit4] 2> 1739156 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!32 (1437860485203492864)]} 0 0
[junit4:junit4] 2> 1739157 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!32 (1437860485203492864)]} 0 4
[junit4:junit4] 2> 1739158 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[g!32]} 0 7
[junit4:junit4] 2> 1739161 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[h!33 (1437860485211881472)]} 0 0
[junit4:junit4] 2> 1739171 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!33 (1437860485217124352)]} 0 1
[junit4:junit4] 2> 1739171 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!33 (1437860485217124352)]} 0 5
[junit4:junit4] 2> 1739172 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[h!33]} 0 8
[junit4:junit4] 2> 1739175 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[i!34 (1437860485226561536)]} 0 0
[junit4:junit4] 2> 1739181 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!34 (1437860485229707264)]} 0 0
[junit4:junit4] 2> 1739182 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[i!34 (1437860485229707264)]} 0 4
[junit4:junit4] 2> 1739185 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[j!35 (1437860485237047296)]} 0 0
[junit4:junit4] 2> 1739194 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!35 (1437860485243338752)]} 0 0
[junit4:junit4] 2> 1739195 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!35 (1437860485243338752)]} 0 4
[junit4:junit4] 2> 1739196 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[j!35]} 0 8
[junit4:junit4] 2> 1739199 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[k!36 (1437860485251727360)]} 0 0
[junit4:junit4] 2> 1739205 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!36 (1437860485254873088)]} 0 0
[junit4:junit4] 2> 1739206 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[k!36 (1437860485254873088)]} 0 4
[junit4:junit4] 2> 1739209 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[l!37 (1437860485262213120)]} 0 0
[junit4:junit4] 2> 1739218 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!37 (1437860485267456000)]} 0 0
[junit4:junit4] 2> 1739219 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[l!37 (1437860485267456000)]} 0 5
[junit4:junit4] 2> 1739220 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[l!37]} 0 8
[junit4:junit4] 2> 1739223 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[m!38 (1437860485276893184)]} 0 0
[junit4:junit4] 2> 1739230 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!38 (1437860485280038912)]} 0 1
[junit4:junit4] 2> 1739230 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[m!38 (1437860485280038912)]} 0 4
[junit4:junit4] 2> 1739233 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[n!39 (1437860485287378944)]} 0 0
[junit4:junit4] 2> 1739241 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!39 (1437860485292621824)]} 0 0
[junit4:junit4] 2> 1739242 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[n!39 (1437860485292621824)]} 0 4
[junit4:junit4] 2> 1739243 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[n!39]} 0 7
[junit4:junit4] 2> 1739246 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[o!40 (1437860485301010432)]} 0 0
[junit4:junit4] 2> 1739252 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!40 (1437860485304156160)]} 0 0
[junit4:junit4] 2> 1739253 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[o!40 (1437860485304156160)]} 0 4
[junit4:junit4] 2> 1739256 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[p!41 (1437860485311496192)]} 0 0
[junit4:junit4] 2> 1739262 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!41 (1437860485314641920)]} 0 0
[junit4:junit4] 2> 1739263 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[p!41 (1437860485314641920)]} 0 4
[junit4:junit4] 2> 1739267 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[q!42 (1437860485323030528)]} 0 0
[junit4:junit4] 2> 1739276 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!42 (1437860485328273408)]} 0 1
[junit4:junit4] 2> 1739277 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[q!42 (1437860485328273408)]} 0 5
[junit4:junit4] 2> 1739277 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[q!42]} 0 7
[junit4:junit4] 2> 1739280 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[r!43 (1437860485336662016)]} 0 0
[junit4:junit4] 2> 1739286 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!43 (1437860485339807744)]} 0 0
[junit4:junit4] 2> 1739287 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[r!43 (1437860485339807744)]} 0 4
[junit4:junit4] 2> 1739291 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[s!44 (1437860485348196352)]} 0 0
[junit4:junit4] 2> 1739299 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!44 (1437860485353439232)]} 0 0
[junit4:junit4] 2> 1739300 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[s!44 (1437860485353439232)]} 0 4
[junit4:junit4] 2> 1739301 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[s!44]} 0 7
[junit4:junit4] 2> 1739304 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[t!45 (1437860485361827840)]} 0 0
[junit4:junit4] 2> 1739313 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!45 (1437860485367070720)]} 0 0
[junit4:junit4] 2> 1739314 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[t!45 (1437860485367070720)]} 0 5
[junit4:junit4] 2> 1739315 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[t!45]} 0 8
[junit4:junit4] 2> 1739318 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[u!46 (1437860485376507904)]} 0 0
[junit4:junit4] 2> 1739325 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!46 (1437860485379653632)]} 0 0
[junit4:junit4] 2> 1739326 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[u!46 (1437860485379653632)]} 0 5
[junit4:junit4] 2> 1739329 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[v!47 (1437860485388042240)]} 0 0
[junit4:junit4] 2> 1739336 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!47 (1437860485391187968)]} 0 1
[junit4:junit4] 2> 1739336 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[v!47 (1437860485391187968)]} 0 4
[junit4:junit4] 2> 1739339 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[w!48 (1437860485398528000)]} 0 0
[junit4:junit4] 2> 1739346 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!48 (1437860485401673728)]} 0 1
[junit4:junit4] 2> 1739346 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[w!48 (1437860485401673728)]} 0 4
[junit4:junit4] 2> 1739350 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[x!49 (1437860485409013760)]} 0 1
[junit4:junit4] 2> 1739357 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!49 (1437860485414256640)]} 0 0
[junit4:junit4] 2> 1739358 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[x!49 (1437860485414256640)]} 0 4
[junit4:junit4] 2> 1739359 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[x!49]} 0 7
[junit4:junit4] 2> 1739363 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[y!50 (1437860485422645248)]} 0 1
[junit4:junit4] 2> 1739371 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!50 (1437860485427888128)]} 0 0
[junit4:junit4] 2> 1739371 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[y!50 (1437860485427888128)]} 0 4
[junit4:junit4] 2> 1739372 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[y!50]} 0 7
[junit4:junit4] 2> 1739375 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[z!51 (1437860485436276736)]} 0 0
[junit4:junit4] 2> 1739382 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!51 (1437860485439422464)]} 0 0
[junit4:junit4] 2> 1739382 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[z!51 (1437860485439422464)]} 0 4
[junit4:junit4] 2> 1739385 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[a!52 (1437860485446762496)]} 0 0
[junit4:junit4] 2> 1739394 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!52 (1437860485452005376)]} 0 1
[junit4:junit4] 2> 1739394 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!52 (1437860485452005376)]} 0 4
[junit4:junit4] 2> 1739395 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[a!52]} 0 7
[junit4:junit4] 2> 1739400 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[b!53 (1437860485461442560)]} 0 1
[junit4:junit4] 2> 1739407 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!53 (1437860485466685440)]} 0 0
[junit4:junit4] 2> 1739408 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[b!53 (1437860485466685440)]} 0 4
[junit4:junit4] 2> 1739409 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[b!53]} 0 7
[junit4:junit4] 2> 1739413 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[c!54 (1437860485476122624)]} 0 0
[junit4:junit4] 2> 1739419 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!54 (1437860485479268352)]} 0 0
[junit4:junit4] 2> 1739420 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[c!54 (1437860485479268352)]} 0 4
[junit4:junit4] 2> 1739423 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[d!55 (1437860485486608384)]} 0 0
[junit4:junit4] 2> 1739432 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!55 (1437860485491851264)]} 0 1
[junit4:junit4] 2> 1739432 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[d!55 (1437860485491851264)]} 0 4
[junit4:junit4] 2> 1739433 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[d!55]} 0 7
[junit4:junit4] 2> 1739436 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[e!56 (1437860485500239872)]} 0 0
[junit4:junit4] 2> 1739443 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!56 (1437860485503385600)]} 0 1
[junit4:junit4] 2> 1739443 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[e!56 (1437860485503385600)]} 0 4
[junit4:junit4] 2> 1739446 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[f!57 (1437860485510725632)]} 0 0
[junit4:junit4] 2> 1739455 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!57 (1437860485515968512)]} 0 1
[junit4:junit4] 2> 1739455 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!57 (1437860485515968512)]} 0 4
[junit4:junit4] 2> 1739456 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[f!57]} 0 7
[junit4:junit4] 2> 1739460 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[g!58 (1437860485524357120)]} 0 1
[junit4:junit4] 2> 1739470 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!58 (1437860485531697152)]} 0 1
[junit4:junit4] 2> 1739471 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!58 (1437860485531697152)]} 0 5
[junit4:junit4] 2> 1739471 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[g!58]} 0 8
[junit4:junit4] 2> 1739475 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[h!59 (1437860485540085760)]} 0 1
[junit4:junit4] 2> 1739482 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!59 (1437860485544280064)]} 0 1
[junit4:junit4] 2> 1739482 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[h!59 (1437860485544280064)]} 0 4
[junit4:junit4] 2> 1739486 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[i!60 (1437860485552668672)]} 0 0
[junit4:junit4] 2> 1739495 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!60 (1437860485557911552)]} 0 1
[junit4:junit4] 2> 1739495 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!60 (1437860485557911552)]} 0 4
[junit4:junit4] 2> 1739496 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[i!60]} 0 7
[junit4:junit4] 2> 1739499 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[j!61 (1437860485566300160)]} 0 0
[junit4:junit4] 2> 1739507 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!61 (1437860485571543040)]} 0 0
[junit4:junit4] 2> 1739508 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!61 (1437860485571543040)]} 0 4
[junit4:junit4] 2> 1739509 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[j!61]} 0 7
[junit4:junit4] 2> 1739512 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[k!62 (1437860485579931648)]} 0 0
[junit4:junit4] 2> 1739518 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!62 (1437860485583077376)]} 0 0
[junit4:junit4] 2> 1739519 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[k!62 (1437860485583077376)]} 0 4
[junit4:junit4] 2> 1739522 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[l!63 (1437860485590417408)]} 0 0
[junit4:junit4] 2> 1739531 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!63 (1437860485595660288)]} 0 0
[junit4:junit4] 2> 1739532 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[l!63 (1437860485595660288)]} 0 5
[junit4:junit4] 2> 1739533 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[l!63]} 0 8
[junit4:junit4] 2> 1739536 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[m!64 (1437860485605097472)]} 0 0
[junit4:junit4] 2> 1739544 T5095 C1099 P10474 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!64 (1437860485610340352)]} 0 0
[junit4:junit4] 2> 1739545 T5068 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[m!64 (1437860485610340352)]} 0 4
[junit4:junit4] 2> 1739546 T5052 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[m!64]} 0 7
[junit4:junit4] 2> 1739550 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[n!65 (1437860485618728960)]} 0 1
[junit4:junit4] 2> 1739558 T5080 C1103 P21316 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:43697/ui/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!65 (1437860485625020416)]} 0 0
[junit4:junit4] 2> 1739559 T5050 C1102 P43697 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={distrib.from=http://127.0.0.1:33218/ui/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[n!65 (1437860485625020416)]} 0 4
[junit4:junit4] 2> 1739560 T5066 C1100 P33218 oasup.LogUpdateProcessor.finish [collection1] webapp=/ui path=/update params={wt=javabin&version=2} {add=[n!65]} 0 8
[junit4:junit4] 2> 1739563 T5026 C1101 P42036 oasup.LogUpdateProcessor.finish [collection1]
[...truncated too long message...]
> 1957536 T5088 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3
[junit4:junit4] 2> 1957536 T5088 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index;done=false>>]
[junit4:junit4] 2> 1957537 T5088 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371250583808/jetty3/index
[junit4:junit4] 2> 1957537 T5088 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1957537 T5088 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/89866278260965386-4-n_0000000001
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:260)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1957538 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1957538 T5088 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1957539 T5088 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1957539 T5088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1957539 T5088 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1957539 T5088 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {id=PostingsFormat(name=NestedPulsing), _version_=PostingsFormat(name=Asserting), n_ti=Pulsing41(freqCutoff=5 minBlockSize=9 maxBlockSize=75)}, docValues:{}, sim=DefaultSimilarity, locale=zh, timezone=Asia/Magadan
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=124185784,total=483393536
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestDocumentBuilder, SpellingQueryConverterTest, CSVRequestHandlerTest, PluginInfoTest, TestGroupingSearch, TestXIncludeConfig, SpatialFilterTest, FastVectorHighlighterTest, TestManagedSchema, BadCopyFieldTest, WordBreakSolrSpellCheckerTest, TestSolrIndexConfig, SOLR749Test, BasicDistributedZkTest, TermVectorComponentTest, AnalysisAfterCoreReloadTest, UnloadDistributedZkTest, TestReloadAndDeleteDocs, SliceStateTest, ShardRoutingTest, CircularListTest, SolrCoreTest, TestFastLRUCache, TestLFUCache, TestTrie, TestRemoteStreaming, TestPropInjectDefaults, LoggingHandlerTest, TestDynamicFieldResource, TestWriterPerf, TestFieldCollectionResource, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, SystemInfoHandlerTest, DistributedQueryElevationComponentTest, OverseerCollectionProcessorTest, RAMDirectoryFactoryTest, ScriptEngineTest, SolrIndexSplitterTest, AlternateDirectoryTest, TestStressVersions, ZkControllerTest, UUIDFieldTest, TestSchemaSimilarityResource, TestFastWriter, PeerSyncTest, TestSearchPerf, TestSolrCoreProperties, CachingDirectoryFactoryTest, TestBadConfig, SchemaVersionSpecificBehaviorTest, TestLRUCache, TestIBSimilarityFactory, BinaryUpdateRequestHandlerTest, UpdateRequestProcessorFactoryTest, TestCharFilters, QueryResultKeyTest, TestCloudManagedSchema, DistributedSpellCheckComponentTest, SoftAutoCommitTest, PrimUtilsTest, SyncSliceTest, TestSerializedLuceneMatchVersion, MBeansHandlerTest, IndexSchemaTest, TestFunctionQuery, ShowFileRequestHandlerTest, TestQuerySenderListener, DirectSolrConnectionTest, StatelessScriptUpdateProcessorFactoryTest, TestReplicationHandler, BadComponentTest, OverseerTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, SliceStateUpdateTest, AutoCommitTest, TestFaceting, TestRecovery, TestRealTimeGet, TestJoin, QueryElevationComponentTest, DirectUpdateHandlerTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SuggesterFSTTest, SolrRequestParserTest, DocValuesTest, SuggesterWFSTTest, PolyFieldTest, TestPseudoReturnFields, QueryEqualityTest, DocValuesMultiTest, TestSolrDeletionPolicy1, CacheHeaderTest, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, RegexBoostProcessorTest, ReturnFieldsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, TestBinaryResponseWriter, IndexReaderFactoryTest, ResponseLogComponentTest, SolrIndexConfigTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, SampleTest, TestDocSet, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, SynonymTokenizerTest, EchoParamsTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, TestPluginEnable, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, TestSolrXMLSerializer, PreAnalyzedFieldTest, DateFieldTest, ClusterStateTest, TestSolrJ, TestRTGBase, SolrTestCaseJ4Test, AliasIntegrationTest, ShardSplitTest]
[junit4:junit4] Completed on J0 in 250.66s, 1 test, 1 error <<< FAILURES!
[...truncated 22 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:385: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:365: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 1 error, 13 ignored (7 assumptions)
Total time: 54 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4065 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4065/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:56766/yl_kf/f returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:56766/yl_kf/f returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([8D5FD34B26C5C88B:CB95D53519AA8B7]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9603 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ShardSplitTest
[junit4:junit4] 2> 964110 T2802 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /yl_kf/f
[junit4:junit4] 2> 964117 T2802 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./solrtest-ShardSplitTest-1371220474146
[junit4:junit4] 2> 964118 T2802 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 964119 T2803 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 964220 T2802 oasc.ZkTestServer.run start zk server on port:56759
[junit4:junit4] 2> 964221 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964224 T2809 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48ab91ed name:ZooKeeperConnection Watcher:127.0.0.1:56759 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964225 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964225 T2802 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 964243 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964245 T2811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45b5ff65 name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964245 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964246 T2802 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 964250 T2802 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 964258 T2802 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 964261 T2802 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 964263 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 964264 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 964275 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 964276 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 964380 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 964381 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 964391 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 964392 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 964395 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 964395 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 964399 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 964400 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 964403 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 964403 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 964406 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 964407 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 964410 T2802 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 964411 T2802 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 964675 T2802 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 964680 T2802 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56762
[junit4:junit4] 2> 964681 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 964682 T2802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 964682 T2802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443
[junit4:junit4] 2> 964683 T2802 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/solr.xml
[junit4:junit4] 2> 964683 T2802 oasc.CoreContainer.<init> New CoreContainer 2063352206
[junit4:junit4] 2> 964684 T2802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/'
[junit4:junit4] 2> 964685 T2802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/'
[junit4:junit4] 2> 964811 T2802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 964812 T2802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 964813 T2802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 964813 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 964814 T2802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 964814 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 964815 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 964815 T2802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 964816 T2802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 964816 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 964834 T2802 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 964835 T2802 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56759/solr
[junit4:junit4] 2> 964835 T2802 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 964836 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964839 T2822 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78437b97 name:ZooKeeperConnection Watcher:127.0.0.1:56759 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964839 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964842 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 964854 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 964856 T2824 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d8c19f0 name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 964856 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 964858 T2802 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 964869 T2802 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 964872 T2802 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 964880 T2802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56762_yl_kf%2Ff
[junit4:junit4] 2> 964882 T2802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56762_yl_kf%2Ff
[junit4:junit4] 2> 964885 T2802 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 964905 T2802 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 964908 T2802 oasc.Overseer.start Overseer (id=89864304994156547-127.0.0.1:56762_yl_kf%2Ff-n_0000000000) starting
[junit4:junit4] 2> 964921 T2802 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 964932 T2826 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 964933 T2802 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 964935 T2802 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 964937 T2802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 964940 T2825 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 964944 T2827 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 964944 T2827 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 966443 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 966444 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56762_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56762/yl_kf/f"}
[junit4:junit4] 2> 966444 T2825 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 966445 T2825 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 966462 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 966947 T2827 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/collection1
[junit4:junit4] 2> 966947 T2827 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 966948 T2827 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 966948 T2827 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 966949 T2827 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/collection1/'
[junit4:junit4] 2> 966951 T2827 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 966951 T2827 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/collection1/lib/README' to classloader
[junit4:junit4] 2> 967003 T2827 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 967063 T2827 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 967164 T2827 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 967169 T2827 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 967745 T2827 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 967749 T2827 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 967752 T2827 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 967757 T2827 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 967780 T2827 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 967781 T2827 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371220474443/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/control/data/
[junit4:junit4] 2> 967781 T2827 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@695d7d60
[junit4:junit4] 2> 967781 T2827 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 967782 T2827 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/control/data
[junit4:junit4] 2> 967782 T2827 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371220474145/control/data/index/
[junit4:junit4] 2> 967783 T2827 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371220474145/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 967783 T2827 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/control/data/index
[junit4:junit4] 2> 967788 T2827 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@262582b5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@75d34698),segFN=segments_1,generation=1}
[junit4:junit4] 2> 967788 T2827 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 967791 T2827 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 967791 T2827 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 967792 T2827 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 967793 T2827 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 967793 T2827 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 967794 T2827 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 967794 T2827 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 967794 T2827 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 967795 T2827 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 967811 T2827 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 967818 T2827 oass.SolrIndexSearcher.<init> Opening Searcher@61874006 main
[junit4:junit4] 2> 967819 T2827 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/control/data/tlog
[junit4:junit4] 2> 967820 T2827 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 967820 T2827 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 967824 T2828 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@61874006 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 967826 T2827 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 967826 T2827 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56762/yl_kf/f collection:control_collection shard:shard1
[junit4:junit4] 2> 967827 T2827 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 967848 T2827 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 967859 T2827 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 967860 T2827 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 967860 T2827 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56762/yl_kf/f/collection1/
[junit4:junit4] 2> 967860 T2827 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 967860 T2827 oasc.SyncStrategy.syncToMe http://127.0.0.1:56762/yl_kf/f/collection1/ has no replicas
[junit4:junit4] 2> 967861 T2827 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56762/yl_kf/f/collection1/
[junit4:junit4] 2> 967861 T2827 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 967966 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 967987 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 968028 T2827 oasc.ZkController.register We are http://127.0.0.1:56762/yl_kf/f/collection1/ and leader is http://127.0.0.1:56762/yl_kf/f/collection1/
[junit4:junit4] 2> 968028 T2827 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56762/yl_kf/f
[junit4:junit4] 2> 968028 T2827 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 968028 T2827 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 968029 T2827 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 968031 T2827 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 968032 T2802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 968032 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 968033 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 968051 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 968052 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 968054 T2831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67b05ab3 name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 968055 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 968056 T2802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 968058 T2802 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 968317 T2802 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 968320 T2802 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56766
[junit4:junit4] 2> 968321 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 968322 T2802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 968322 T2802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086
[junit4:junit4] 2> 968323 T2802 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/solr.xml
[junit4:junit4] 2> 968323 T2802 oasc.CoreContainer.<init> New CoreContainer 1867519919
[junit4:junit4] 2> 968324 T2802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/'
[junit4:junit4] 2> 968325 T2802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/'
[junit4:junit4] 2> 968451 T2802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 968452 T2802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 968452 T2802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 968453 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 968453 T2802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 968454 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 968454 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 968455 T2802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 968455 T2802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 968456 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 968473 T2802 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 968474 T2802 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56759/solr
[junit4:junit4] 2> 968474 T2802 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 968475 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 968479 T2842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2266495a name:ZooKeeperConnection Watcher:127.0.0.1:56759 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 968479 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 968493 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 968505 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 968507 T2844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d36381b name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 968508 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 968519 T2802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 969501 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 969502 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56762_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56762/yl_kf/f"}
[junit4:junit4] 2> 969523 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 969524 T2802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56766_yl_kf%2Ff
[junit4:junit4] 2> 969523 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 969523 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 969532 T2802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56766_yl_kf%2Ff
[junit4:junit4] 2> 969534 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 969534 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 969534 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 969536 T2831 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 969537 T2824 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 969537 T2844 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 969543 T2845 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 969543 T2845 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 971034 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 971035 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56766_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56766/yl_kf/f"}
[junit4:junit4] 2> 971035 T2825 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 971035 T2825 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 971051 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 971052 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 971052 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 971547 T2845 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/collection1
[junit4:junit4] 2> 971547 T2845 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 971548 T2845 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 971548 T2845 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 971550 T2845 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/collection1/'
[junit4:junit4] 2> 971551 T2845 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/collection1/lib/README' to classloader
[junit4:junit4] 2> 971552 T2845 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 971604 T2845 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 971663 T2845 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 971765 T2845 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 971771 T2845 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 972347 T2845 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 972351 T2845 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 972354 T2845 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 972359 T2845 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 972383 T2845 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 972384 T2845 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371220478086/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty1/
[junit4:junit4] 2> 972384 T2845 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@695d7d60
[junit4:junit4] 2> 972385 T2845 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 972385 T2845 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty1
[junit4:junit4] 2> 972386 T2845 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty1/index/
[junit4:junit4] 2> 972386 T2845 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 972387 T2845 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty1/index
[junit4:junit4] 2> 972390 T2845 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4a8abce7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@47f9587f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 972391 T2845 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 972394 T2845 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 972394 T2845 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 972395 T2845 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 972396 T2845 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 972396 T2845 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 972397 T2845 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 972397 T2845 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 972398 T2845 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 972398 T2845 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 972415 T2845 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 972422 T2845 oass.SolrIndexSearcher.<init> Opening Searcher@a816338 main
[junit4:junit4] 2> 972422 T2845 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty1/tlog
[junit4:junit4] 2> 972424 T2845 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 972424 T2845 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 972428 T2846 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a816338 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 972430 T2845 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 972431 T2845 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56766/yl_kf/f collection:collection1 shard:shard1
[junit4:junit4] 2> 972431 T2845 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 972530 T2845 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 972601 T2845 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 972601 T2845 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 972601 T2845 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56766/yl_kf/f/collection1/
[junit4:junit4] 2> 972601 T2845 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 972602 T2845 oasc.SyncStrategy.syncToMe http://127.0.0.1:56766/yl_kf/f/collection1/ has no replicas
[junit4:junit4] 2> 972602 T2845 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56766/yl_kf/f/collection1/
[junit4:junit4] 2> 972602 T2845 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 974102 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 974125 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 974125 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 974125 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 974181 T2845 oasc.ZkController.register We are http://127.0.0.1:56766/yl_kf/f/collection1/ and leader is http://127.0.0.1:56766/yl_kf/f/collection1/
[junit4:junit4] 2> 974181 T2845 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56766/yl_kf/f
[junit4:junit4] 2> 974181 T2845 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 974181 T2845 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 974182 T2845 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 974183 T2845 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 974184 T2802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 974185 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 974186 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 974463 T2802 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 974466 T2802 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56772
[junit4:junit4] 2> 974467 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 974468 T2802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 974468 T2802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229
[junit4:junit4] 2> 974469 T2802 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/solr.xml
[junit4:junit4] 2> 974469 T2802 oasc.CoreContainer.<init> New CoreContainer 688305908
[junit4:junit4] 2> 974470 T2802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/'
[junit4:junit4] 2> 974471 T2802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/'
[junit4:junit4] 2> 974602 T2802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 974603 T2802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 974603 T2802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 974604 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 974604 T2802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 974605 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 974605 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 974606 T2802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 974606 T2802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 974607 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 974623 T2802 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 974623 T2802 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56759/solr
[junit4:junit4] 2> 974624 T2802 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 974625 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 974628 T2858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b842aac name:ZooKeeperConnection Watcher:127.0.0.1:56759 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 974628 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 974631 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 974642 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 974644 T2860 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7747bd0d name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 974644 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 974656 T2802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 975638 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 975639 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56766_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56766/yl_kf/f"}
[junit4:junit4] 2> 975657 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975657 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975657 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975657 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975659 T2802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56772_yl_kf%2Ff
[junit4:junit4] 2> 975661 T2802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56772_yl_kf%2Ff
[junit4:junit4] 2> 975669 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975669 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975669 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975669 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 975671 T2860 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 975671 T2831 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 975672 T2824 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 975672 T2844 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 975677 T2861 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 975677 T2861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 977163 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 977164 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56772_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56772/yl_kf/f"}
[junit4:junit4] 2> 977165 T2825 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 977165 T2825 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 977181 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 977182 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 977182 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 977182 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 977681 T2861 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/collection1
[junit4:junit4] 2> 977681 T2861 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 977682 T2861 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 977682 T2861 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 977684 T2861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/collection1/'
[junit4:junit4] 2> 977685 T2861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/collection1/lib/README' to classloader
[junit4:junit4] 2> 977686 T2861 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 977738 T2861 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 977797 T2861 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 977898 T2861 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 977904 T2861 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 978476 T2861 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 978480 T2861 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 978483 T2861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 978488 T2861 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 978512 T2861 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 978512 T2861 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371220484229/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty2/
[junit4:junit4] 2> 978512 T2861 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@695d7d60
[junit4:junit4] 2> 978513 T2861 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 978515 T2861 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty2
[junit4:junit4] 2> 978515 T2861 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty2/index/
[junit4:junit4] 2> 978515 T2861 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 978516 T2861 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty2/index
[junit4:junit4] 2> 978519 T2861 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d54ab9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74dc350e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 978520 T2861 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 978523 T2861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 978523 T2861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 978524 T2861 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 978525 T2861 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 978525 T2861 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 978525 T2861 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 978526 T2861 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 978527 T2861 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 978527 T2861 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 978543 T2861 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 978551 T2861 oass.SolrIndexSearcher.<init> Opening Searcher@10b720bd main
[junit4:junit4] 2> 978551 T2861 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty2/tlog
[junit4:junit4] 2> 978552 T2861 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 978553 T2861 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 978557 T2862 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10b720bd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 978560 T2861 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 978560 T2861 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56772/yl_kf/f collection:collection1 shard:shard2
[junit4:junit4] 2> 978561 T2861 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 978580 T2861 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 978584 T2861 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 978584 T2861 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 978584 T2861 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56772/yl_kf/f/collection1/
[junit4:junit4] 2> 978585 T2861 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 978585 T2861 oasc.SyncStrategy.syncToMe http://127.0.0.1:56772/yl_kf/f/collection1/ has no replicas
[junit4:junit4] 2> 978585 T2861 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56772/yl_kf/f/collection1/
[junit4:junit4] 2> 978585 T2861 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 978687 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 978704 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 978704 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 978704 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 978704 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 978752 T2861 oasc.ZkController.register We are http://127.0.0.1:56772/yl_kf/f/collection1/ and leader is http://127.0.0.1:56772/yl_kf/f/collection1/
[junit4:junit4] 2> 978752 T2861 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56772/yl_kf/f
[junit4:junit4] 2> 978752 T2861 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 978752 T2861 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 978753 T2861 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 978754 T2861 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 978756 T2802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 978756 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 978757 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 979033 T2802 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 979036 T2802 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56787
[junit4:junit4] 2> 979037 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 979037 T2802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 979038 T2802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800
[junit4:junit4] 2> 979038 T2802 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/solr.xml
[junit4:junit4] 2> 979039 T2802 oasc.CoreContainer.<init> New CoreContainer 172387983
[junit4:junit4] 2> 979040 T2802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/'
[junit4:junit4] 2> 979040 T2802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/'
[junit4:junit4] 2> 979166 T2802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 979167 T2802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 979168 T2802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 979168 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 979169 T2802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 979169 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 979170 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 979170 T2802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 979171 T2802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 979171 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 979187 T2802 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 979187 T2802 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56759/solr
[junit4:junit4] 2> 979188 T2802 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 979189 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 979228 T2874 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@515f40f1 name:ZooKeeperConnection Watcher:127.0.0.1:56759 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 979229 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 979239 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 979249 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 979251 T2876 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34087b55 name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 979251 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 979256 T2802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 980217 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 980218 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56772_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56772/yl_kf/f"}
[junit4:junit4] 2> 980232 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980232 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980232 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980232 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980232 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980259 T2802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56787_yl_kf%2Ff
[junit4:junit4] 2> 980261 T2802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56787_yl_kf%2Ff
[junit4:junit4] 2> 980263 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980263 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980263 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980263 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980263 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 980264 T2860 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 980265 T2831 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 980266 T2844 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 980266 T2824 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 980266 T2876 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 980271 T2877 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 980271 T2877 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 981737 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 981738 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56787_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56787/yl_kf/f"}
[junit4:junit4] 2> 981739 T2825 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 981739 T2825 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 981754 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 981754 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 981754 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 981754 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 981754 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 982275 T2877 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/collection1
[junit4:junit4] 2> 982275 T2877 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 982276 T2877 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 982276 T2877 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 982278 T2877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/collection1/'
[junit4:junit4] 2> 982279 T2877 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/collection1/lib/README' to classloader
[junit4:junit4] 2> 982280 T2877 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 982334 T2877 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 982395 T2877 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 982497 T2877 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 982503 T2877 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 983075 T2877 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 983078 T2877 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 983080 T2877 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 983084 T2877 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 983104 T2877 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 983104 T2877 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371220488800/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/
[junit4:junit4] 2> 983105 T2877 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@695d7d60
[junit4:junit4] 2> 983105 T2877 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 983106 T2877 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3
[junit4:junit4] 2> 983106 T2877 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index/
[junit4:junit4] 2> 983106 T2877 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 983107 T2877 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index
[junit4:junit4] 2> 983110 T2877 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@645ea06d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c1378a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 983110 T2877 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 983112 T2877 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 983112 T2877 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 983113 T2877 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 983113 T2877 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 983114 T2877 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 983114 T2877 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 983114 T2877 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 983115 T2877 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 983115 T2877 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 983126 T2877 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 983130 T2877 oass.SolrIndexSearcher.<init> Opening Searcher@55f73544 main
[junit4:junit4] 2> 983131 T2877 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/tlog
[junit4:junit4] 2> 983132 T2877 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 983132 T2877 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 983135 T2878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55f73544 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 983137 T2877 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 983137 T2877 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56787/yl_kf/f collection:collection1 shard:shard1
[junit4:junit4] 2> 983171 T2877 oasc.ZkController.register We are http://127.0.0.1:56787/yl_kf/f/collection1/ and leader is http://127.0.0.1:56766/yl_kf/f/collection1/
[junit4:junit4] 2> 983171 T2877 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56787/yl_kf/f
[junit4:junit4] 2> 983171 T2877 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 983171 T2877 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C564 name=collection1 org.apache.solr.core.SolrCore@65e04983 url=http://127.0.0.1:56787/yl_kf/f/collection1 node=127.0.0.1:56787_yl_kf%2Ff C564_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56787_yl_kf%2Ff, base_url=http://127.0.0.1:56787/yl_kf/f}
[junit4:junit4] 2> 983172 T2879 C564 P56787 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 983172 T2877 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 983172 T2879 C564 P56787 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 983173 T2879 C564 P56787 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 983173 T2879 C564 P56787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 983173 T2802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 983174 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 983174 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 983174 T2879 C564 P56787 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 983192 T2834 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 983258 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 983259 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56787_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56787/yl_kf/f"}
[junit4:junit4] 2> 983280 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 983280 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 983280 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 983280 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 983280 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 983360 T2802 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 983362 T2802 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56791
[junit4:junit4] 2> 983363 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 983363 T2802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 983363 T2802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218
[junit4:junit4] 2> 983364 T2802 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/solr.xml
[junit4:junit4] 2> 983364 T2802 oasc.CoreContainer.<init> New CoreContainer 2017614680
[junit4:junit4] 2> 983365 T2802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/'
[junit4:junit4] 2> 983365 T2802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/'
[junit4:junit4] 2> 983465 T2802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 983466 T2802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 983466 T2802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 983466 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 983467 T2802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 983467 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 983467 T2802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 983468 T2802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 983468 T2802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 983469 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 983480 T2802 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 983480 T2802 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56759/solr
[junit4:junit4] 2> 983481 T2802 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 983481 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 983485 T2891 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@631a5480 name:ZooKeeperConnection Watcher:127.0.0.1:56759 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 983485 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 983498 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 983510 T2802 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 983512 T2893 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62cfbf04 name:ZooKeeperConnection Watcher:127.0.0.1:56759/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 983512 T2802 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 983517 T2802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 984193 T2834 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 984193 T2834 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:56787_yl_kf%252Ff&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 984520 T2802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56791_yl_kf%2Ff
[junit4:junit4] 2> 984523 T2802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56791_yl_kf%2Ff
[junit4:junit4] 2> 984525 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 984525 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 984526 T2893 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 984525 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 984525 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 984525 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 984527 T2860 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 984528 T2844 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 984538 T2876 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 984539 T2824 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 984539 T2831 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 984543 T2894 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 984544 T2894 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 984785 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 984786 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56791_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56791/yl_kf/f"}
[junit4:junit4] 2> 984786 T2825 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 984786 T2825 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 984802 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 984802 T2893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 984802 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 984802 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 984802 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 984802 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 985546 T2894 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/collection1
[junit4:junit4] 2> 985546 T2894 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 985547 T2894 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 985547 T2894 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 985549 T2894 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/collection1/'
[junit4:junit4] 2> 985550 T2894 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 985551 T2894 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/collection1/lib/README' to classloader
[junit4:junit4] 2> 985602 T2894 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 985662 T2894 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 985764 T2894 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 985770 T2894 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C565 name=collection1 org.apache.solr.core.SolrCore@65e04983 url=http://127.0.0.1:56787/yl_kf/f/collection1 node=127.0.0.1:56787_yl_kf%2Ff C565_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56787_yl_kf%2Ff, base_url=http://127.0.0.1:56787/yl_kf/f}
[junit4:junit4] 2> 986195 T2879 C565 P56787 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:56766/yl_kf/f/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 986195 T2879 C565 P56787 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56787/yl_kf/f START replicas=[http://127.0.0.1:56766/yl_kf/f/collection1/] nUpdates=100
[junit4:junit4] 2> 986195 T2879 C565 P56787 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 986197 T2879 C565 P56787 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 986197 T2879 C565 P56787 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 986198 T2879 C565 P56787 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 986198 T2879 C565 P56787 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 986198 T2879 C565 P56787 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:56766/yl_kf/f/collection1/. core=collection1
[junit4:junit4] 2> 986198 T2879 C565 P56787 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C566 name=collection1 org.apache.solr.core.SolrCore@442bb5e9 url=http://127.0.0.1:56766/yl_kf/f/collection1 node=127.0.0.1:56766_yl_kf%2Ff C566_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56766_yl_kf%2Ff, base_url=http://127.0.0.1:56766/yl_kf/f, leader=true}
[junit4:junit4] 2> 986205 T2835 C566 P56766 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 986210 T2836 C566 P56766 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 986212 T2836 C566 P56766 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4a8abce7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@47f9587f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 986213 T2836 C566 P56766 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 986214 T2836 C566 P56766 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4a8abce7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@47f9587f),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4a8abce7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@47f9587f),segFN=segments_2,generation=2}
[junit4:junit4] 2> 986214 T2836 C566 P56766 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 986215 T2836 C566 P56766 oass.SolrIndexSearcher.<init> Opening Searcher@3d32d7d5 realtime
[junit4:junit4] 2> 986215 T2836 C566 P56766 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 986216 T2836 C566 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 986217 T2879 C565 P56787 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 986217 T2879 C565 P56787 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 986219 T2837 C566 P56766 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 986220 T2837 C566 P56766 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 986220 T2879 C565 P56787 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 986220 T2879 C565 P56787 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 986221 T2879 C565 P56787 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 986223 T2837 C566 P56766 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 986224 T2879 C565 P56787 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 986225 T2879 C565 P56787 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index.20130614083456252
[junit4:junit4] 2> 986225 T2879 C565 P56787 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4b1d4d16 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6f1ce2c9) fullCopy=false
[junit4:junit4] 2> 986228 T2837 C566 P56766 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C567 name=collection1 org.apache.solr.core.SolrCore@65e04983 url=http://127.0.0.1:56787/yl_kf/f/collection1 node=127.0.0.1:56787_yl_kf%2Ff C567_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56787_yl_kf%2Ff, base_url=http://127.0.0.1:56787/yl_kf/f}
[junit4:junit4] 2> 986247 T2879 C567 P56787 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 986248 T2879 C567 P56787 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 986248 T2879 C567 P56787 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 986250 T2879 C567 P56787 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@645ea06d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c1378a),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@645ea06d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c1378a),segFN=segments_2,generation=2}
[junit4:junit4] 2> 986251 T2879 C567 P56787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 986251 T2879 C567 P56787 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 986252 T2879 C567 P56787 oass.SolrIndexSearcher.<init> Opening Searcher@43f9da03 main
[junit4:junit4] 2> 986253 T2878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43f9da03 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 986254 T2879 C567 P56787 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index.20130614083456252 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index.20130614083456252;done=true>>]
[junit4:junit4] 2> 986254 T2879 C567 P56787 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index.20130614083456252
[junit4:junit4] 2> 986254 T2879 C567 P56787 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty3/index.20130614083456252
[junit4:junit4] 2> 986254 T2879 C567 P56787 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 986255 T2879 C567 P56787 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 986255 T2879 C567 P56787 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 986255 T2879 C567 P56787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 986257 T2879 C567 P56787 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 986362 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 986363 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56787_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56787/yl_kf/f"}
[junit4:junit4] 2> 986376 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 986376 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 986377 T2893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 986376 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 986376 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 986376 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 986411 T2894 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 986415 T2894 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 986417 T2894 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 986422 T2894 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 986446 T2894 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 986446 T2894 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371220493218/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/
[junit4:junit4] 2> 986446 T2894 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@695d7d60
[junit4:junit4] 2> 986447 T2894 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 986448 T2894 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4
[junit4:junit4] 2> 986448 T2894 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index/
[junit4:junit4] 2> 986448 T2894 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 986449 T2894 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index
[junit4:junit4] 2> 986453 T2894 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 986453 T2894 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 986456 T2894 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 986457 T2894 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 986457 T2894 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 986458 T2894 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 986459 T2894 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 986459 T2894 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 986460 T2894 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 986461 T2894 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 986461 T2894 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 986477 T2894 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 986484 T2894 oass.SolrIndexSearcher.<init> Opening Searcher@dd9e531 main
[junit4:junit4] 2> 986485 T2894 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/tlog
[junit4:junit4] 2> 986486 T2894 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 986486 T2894 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 986491 T2897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@dd9e531 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 986493 T2894 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 986493 T2894 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56791/yl_kf/f collection:collection1 shard:shard2
[junit4:junit4] 2> 986497 T2894 oasc.ZkController.register We are http://127.0.0.1:56791/yl_kf/f/collection1/ and leader is http://127.0.0.1:56772/yl_kf/f/collection1/
[junit4:junit4] 2> 986497 T2894 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56791/yl_kf/f
[junit4:junit4] 2> 986497 T2894 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 986497 T2894 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C568 name=collection1 org.apache.solr.core.SolrCore@7c576c0a url=http://127.0.0.1:56791/yl_kf/f/collection1 node=127.0.0.1:56791_yl_kf%2Ff C568_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56791_yl_kf%2Ff, base_url=http://127.0.0.1:56791/yl_kf/f}
[junit4:junit4] 2> 986498 T2898 C568 P56791 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 986498 T2894 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 986498 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 986499 T2898 C568 P56791 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 986499 T2898 C568 P56791 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 986500 T2802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 986500 T2802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 986500 T2898 C568 P56791 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 986501 T2802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 986519 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 986520 T2850 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 986520 T2802 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 986521 T2802 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 986522 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 987524 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 987881 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 987882 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56791_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56791/yl_kf/f"}
[junit4:junit4] 2> 987908 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 987908 T2893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 987908 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 987908 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 987908 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 987908 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 988522 T2850 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 988522 T2850 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:56791_yl_kf%252Ff&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 988527 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 989529 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C568_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56791_yl_kf%2Ff, base_url=http://127.0.0.1:56791/yl_kf/f}
[junit4:junit4] 2> 990524 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:56772/yl_kf/f/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 990524 T2898 C568 P56791 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56791/yl_kf/f START replicas=[http://127.0.0.1:56772/yl_kf/f/collection1/] nUpdates=100
[junit4:junit4] 2> 990525 T2898 C568 P56791 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 990525 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 990525 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 990525 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 990526 T2898 C568 P56791 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 990526 T2898 C568 P56791 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:56772/yl_kf/f/collection1/. core=collection1
[junit4:junit4] 2> 990526 T2898 C568 P56791 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 990531 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C569 name=collection1 org.apache.solr.core.SolrCore@18438c0f url=http://127.0.0.1:56772/yl_kf/f/collection1 node=127.0.0.1:56772_yl_kf%2Ff C569_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56772_yl_kf%2Ff, base_url=http://127.0.0.1:56772/yl_kf/f, leader=true}
[junit4:junit4] 2> 990534 T2851 C569 P56772 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 990538 T2852 C569 P56772 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 990540 T2852 C569 P56772 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d54ab9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74dc350e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 990541 T2852 C569 P56772 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 990541 T2852 C569 P56772 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d54ab9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74dc350e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d54ab9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74dc350e),segFN=segments_2,generation=2}
[junit4:junit4] 2> 990542 T2852 C569 P56772 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 990542 T2852 C569 P56772 oass.SolrIndexSearcher.<init> Opening Searcher@53ddc315 realtime
[junit4:junit4] 2> 990543 T2852 C569 P56772 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 990543 T2852 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 990544 T2898 C568 P56791 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 990544 T2898 C568 P56791 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 990546 T2853 C569 P56772 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 990546 T2853 C569 P56772 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 990547 T2898 C568 P56791 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 990547 T2898 C568 P56791 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 990547 T2898 C568 P56791 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 990550 T2853 C569 P56772 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 990551 T2898 C568 P56791 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 990552 T2898 C568 P56791 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index.20130614083500579
[junit4:junit4] 2> 990552 T2898 C568 P56791 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e3b4ad2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1dc23e89) fullCopy=false
[junit4:junit4] 2> 990555 T2853 C569 P56772 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 990556 T2898 C568 P56791 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 990557 T2898 C568 P56791 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 990557 T2898 C568 P56791 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 990559 T2898 C568 P56791 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_2,generation=2}
[junit4:junit4] 2> 990559 T2898 C568 P56791 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 990559 T2898 C568 P56791 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 990560 T2898 C568 P56791 oass.SolrIndexSearcher.<init> Opening Searcher@77cdc8e9 main
[junit4:junit4] 2> 990561 T2897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@77cdc8e9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 990562 T2898 C568 P56791 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index.20130614083500579 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index.20130614083500579;done=true>>]
[junit4:junit4] 2> 990562 T2898 C568 P56791 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index.20130614083500579
[junit4:junit4] 2> 990562 T2898 C568 P56791 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index.20130614083500579
[junit4:junit4] 2> 990562 T2898 C568 P56791 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 990562 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 990563 T2898 C568 P56791 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 990563 T2898 C568 P56791 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 990565 T2898 C568 P56791 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 990914 T2825 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 990915 T2825 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56791_yl_kf%2Ff",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56791/yl_kf/f"}
[junit4:junit4] 2> 990931 T2860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 990931 T2893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 990931 T2831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 990931 T2876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 990931 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 990931 T2844 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 991533 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 991535 T2802 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C570 name=collection1 org.apache.solr.core.SolrCore@249d97f0 url=http://127.0.0.1:56762/yl_kf/f/collection1 node=127.0.0.1:56762_yl_kf%2Ff C570_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:56762_yl_kf%2Ff, base_url=http://127.0.0.1:56762/yl_kf/f, leader=true}
[junit4:junit4] 2> 991543 T2814 C570 P56762 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 991546 T2814 C570 P56762 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@262582b5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@75d34698),segFN=segments_1,generation=1}
[junit4:junit4] 2> 991547 T2814 C570 P56762 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 991548 T2814 C570 P56762 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@262582b5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@75d34698),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@262582b5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@75d34698),segFN=segments_2,generation=2}
[junit4:junit4] 2> 991548 T2814 C570 P56762 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 991549 T2814 C570 P56762 oass.SolrIndexSearcher.<init> Opening Searcher@2380b4c8 main
[junit4:junit4] 2> 991549 T2814 C570 P56762 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 991550 T2828 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2380b4c8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 991551 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 991561 T2854 C569 P56772 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 991562 T2854 C569 P56772 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d54ab9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74dc350e),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d54ab9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74dc350e),segFN=segments_3,generation=3}
[junit4:junit4] 2> 991563 T2854 C569 P56772 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 991563 T2854 C569 P56772 oass.SolrIndexSearcher.<init> Opening Searcher@7af59a82 main
[junit4:junit4] 2> 991564 T2854 C569 P56772 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 991564 T2862 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7af59a82 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 991565 T2854 C569 P56772 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:56766/yl_kf/f/collection1/, StdNode: http://127.0.0.1:56787/yl_kf/f/collection1/, StdNode: http://127.0.0.1:56791/yl_kf/f/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C571 name=collection1 org.apache.solr.core.SolrCore@442bb5e9 url=http://127.0.0.1:56766/yl_kf/f/collection1 node=127.0.0.1:56766_yl_kf%2Ff C571_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56766_yl_kf%2Ff, base_url=http://127.0.0.1:56766/yl_kf/f, leader=true}
[junit4:junit4] 2> 991568 T2838 C571 P56766 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 991568 T2838 C571 P56766 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4a8abce7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@47f9587f),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4a8abce7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@47f9587f),segFN=segments_3,generation=3}
[junit4:junit4] 2> 991569 T2838 C571 P56766 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 991569 T2838 C571 P56766 oass.SolrIndexSearcher.<init> Opening Searcher@685eaff1 main
[junit4:junit4] 2> 991570 T2838 C571 P56766 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 991571 T2846 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@685eaff1 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 991571 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> C568_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56791_yl_kf%2Ff, base_url=http://127.0.0.1:56791/yl_kf/f}
[junit4:junit4] 2> 991575 T2883 C568 P56791 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C567_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56787_yl_kf%2Ff, base_url=http://127.0.0.1:56787/yl_kf/f}
[junit4:junit4] 2> 991575 T2866 C567 P56787 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 991576 T2883 C568 P56791 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_3,generation=3}
[junit4:junit4] 2> 991576 T2866 C567 P56787 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@645ea06d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c1378a),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@645ea06d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c1378a),segFN=segments_3,generation=3}
[junit4:junit4] 2> 991576 T2883 C568 P56791 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 991577 T2866 C567 P56787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 991578 T2883 C568 P56791 oass.SolrIndexSearcher.<init> Opening Searcher@301a19c3 main
[junit4:junit4] 2> 991578 T2866 C567 P56787 oass.SolrIndexSearcher.<init> Opening Searcher@699f06a9 main
[junit4:junit4] 2> 991579 T2883 C568 P56791 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 991579 T2897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@301a19c3 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 991579 T2866 C567 P56787 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 991580 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 991580 T2878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@699f06a9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 991581 T2866 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 991582 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> 991583 T2802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 991585 T2839 C571 P56766 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 991587 T2867 C567 P56787 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 991589 T2855 C569 P56772 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 991591 T2884 C568 P56791 oasc.SolrCore.execute [collection1] webapp=/yl_kf/f path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 993596 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437828910806990848)} 0 2
[junit4:junit4] 2> 993604 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={update.distrib=FROMLEADER&_version_=-1437828910811185152&update.from=http://127.0.0.1:56772/yl_kf/f/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437828910811185152)} 0 1
[junit4:junit4] 2> 993608 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={update.distrib=FROMLEADER&_version_=-1437828910815379456&update.from=http://127.0.0.1:56766/yl_kf/f/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437828910815379456)} 0 1
[junit4:junit4] 2> 993609 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1437828910815379456)} 0 6
[junit4:junit4] 2> 993610 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437828910811185152)} 0 11
[junit4:junit4] 2> 993614 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[a!0 (1437828910825865216)]} 0 1
[junit4:junit4] 2> 993620 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!0 (1437828910829010944)]} 0 1
[junit4:junit4] 2> 993620 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[a!0 (1437828910829010944)]} 0 4
[junit4:junit4] 2> 993623 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[b!1 (1437828910836350976)]} 0 0
[junit4:junit4] 2> 993629 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!1 (1437828910838448128)]} 0 1
[junit4:junit4] 2> 993629 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[b!1 (1437828910838448128)]} 0 4
[junit4:junit4] 2> 993632 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[c!2 (1437828910845788160)]} 0 0
[junit4:junit4] 2> 993640 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!2 (1437828910851031040)]} 0 0
[junit4:junit4] 2> 993641 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[c!2 (1437828910851031040)]} 0 4
[junit4:junit4] 2> 993642 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[c!2]} 0 7
[junit4:junit4] 2> 993644 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[d!3 (1437828910858371072)]} 0 0
[junit4:junit4] 2> 993650 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!3 (1437828910861516800)]} 0 0
[junit4:junit4] 2> 993651 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[d!3 (1437828910861516800)]} 0 4
[junit4:junit4] 2> 993654 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[e!4 (1437828910868856832)]} 0 0
[junit4:junit4] 2> 993660 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!4 (1437828910872002560)]} 0 0
[junit4:junit4] 2> 993661 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[e!4 (1437828910872002560)]} 0 4
[junit4:junit4] 2> 993664 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[f!5 (1437828910879342592)]} 0 0
[junit4:junit4] 2> 993671 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!5 (1437828910883536896)]} 0 0
[junit4:junit4] 2> 993672 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!5 (1437828910883536896)]} 0 4
[junit4:junit4] 2> 993673 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[f!5]} 0 6
[junit4:junit4] 2> 993675 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[g!6 (1437828910890876928)]} 0 0
[junit4:junit4] 2> 993680 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!6 (1437828910894022656)]} 0 0
[junit4:junit4] 2> 993681 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[g!6 (1437828910894022656)]} 0 3
[junit4:junit4] 2> 993684 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[h!7 (1437828910900314112)]} 0 0
[junit4:junit4] 2> 993689 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!7 (1437828910903459840)]} 0 0
[junit4:junit4] 2> 993690 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[h!7 (1437828910903459840)]} 0 3
[junit4:junit4] 2> 993693 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[i!8 (1437828910909751296)]} 0 0
[junit4:junit4] 2> 993699 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!8 (1437828910914994176)]} 0 0
[junit4:junit4] 2> 993700 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!8 (1437828910914994176)]} 0 2
[junit4:junit4] 2> 993701 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[i!8]} 0 5
[junit4:junit4] 2> 993703 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[j!9 (1437828910920237056)]} 0 0
[junit4:junit4] 2> 993710 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!9 (1437828910925479936)]} 0 0
[junit4:junit4] 2> 993711 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!9 (1437828910925479936)]} 0 3
[junit4:junit4] 2> 993712 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[j!9]} 0 6
[junit4:junit4] 2> 993714 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[k!10 (1437828910931771392)]} 0 0
[junit4:junit4] 2> 993722 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!10 (1437828910937014272)]} 0 0
[junit4:junit4] 2> 993723 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[k!10 (1437828910937014272)]} 0 4
[junit4:junit4] 2> 993724 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[k!10]} 0 7
[junit4:junit4] 2> 993726 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[l!11 (1437828910944354304)]} 0 0
[junit4:junit4] 2> 993734 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!11 (1437828910949597184)]} 0 0
[junit4:junit4] 2> 993735 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[l!11 (1437828910949597184)]} 0 4
[junit4:junit4] 2> 993736 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[l!11]} 0 7
[junit4:junit4] 2> 993738 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[m!12 (1437828910956937216)]} 0 0
[junit4:junit4] 2> 993745 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!12 (1437828910962180096)]} 0 0
[junit4:junit4] 2> 993746 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[m!12 (1437828910962180096)]} 0 3
[junit4:junit4] 2> 993747 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[m!12]} 0 6
[junit4:junit4] 2> 993749 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[n!13 (1437828910968471552)]} 0 0
[junit4:junit4] 2> 993755 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!13 (1437828910971617280)]} 0 0
[junit4:junit4] 2> 993756 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[n!13 (1437828910971617280)]} 0 4
[junit4:junit4] 2> 993759 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[o!14 (1437828910978957312)]} 0 0
[junit4:junit4] 2> 993765 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!14 (1437828910982103040)]} 0 0
[junit4:junit4] 2> 993766 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[o!14 (1437828910982103040)]} 0 4
[junit4:junit4] 2> 993769 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[p!15 (1437828910989443072)]} 0 0
[junit4:junit4] 2> 993777 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!15 (1437828910994685952)]} 0 0
[junit4:junit4] 2> 993778 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[p!15 (1437828910994685952)]} 0 4
[junit4:junit4] 2> 993779 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[p!15]} 0 7
[junit4:junit4] 2> 993781 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[q!16 (1437828911002025984)]} 0 0
[junit4:junit4] 2> 993786 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!16 (1437828911005171712)]} 0 0
[junit4:junit4] 2> 993787 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[q!16 (1437828911005171712)]} 0 3
[junit4:junit4] 2> 993790 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[r!17 (1437828911011463168)]} 0 0
[junit4:junit4] 2> 993796 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!17 (1437828911014608896)]} 0 0
[junit4:junit4] 2> 993797 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[r!17 (1437828911014608896)]} 0 4
[junit4:junit4] 2> 993800 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[s!18 (1437828911021948928)]} 0 0
[junit4:junit4] 2> 993806 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!18 (1437828911025094656)]} 0 0
[junit4:junit4] 2> 993807 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[s!18 (1437828911025094656)]} 0 4
[junit4:junit4] 2> 993810 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[t!19 (1437828911032434688)]} 0 0
[junit4:junit4] 2> 993818 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!19 (1437828911037677568)]} 0 0
[junit4:junit4] 2> 993819 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[t!19 (1437828911037677568)]} 0 4
[junit4:junit4] 2> 993820 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[t!19]} 0 7
[junit4:junit4] 2> 993822 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[u!20 (1437828911045017600)]} 0 0
[junit4:junit4] 2> 993828 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!20 (1437828911048163328)]} 0 0
[junit4:junit4] 2> 993829 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[u!20 (1437828911048163328)]} 0 4
[junit4:junit4] 2> 993832 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[v!21 (1437828911055503360)]} 0 0
[junit4:junit4] 2> 993839 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!21 (1437828911060746240)]} 0 0
[junit4:junit4] 2> 993840 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[v!21 (1437828911060746240)]} 0 3
[junit4:junit4] 2> 993841 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[v!21]} 0 6
[junit4:junit4] 2> 993843 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[w!22 (1437828911067037696)]} 0 0
[junit4:junit4] 2> 993850 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!22 (1437828911072280576)]} 0 0
[junit4:junit4] 2> 993851 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[w!22 (1437828911072280576)]} 0 3
[junit4:junit4] 2> 993852 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[w!22]} 0 6
[junit4:junit4] 2> 993854 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[x!23 (1437828911078572032)]} 0 0
[junit4:junit4] 2> 993861 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!23 (1437828911083814912)]} 0 0
[junit4:junit4] 2> 993862 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[x!23 (1437828911083814912)]} 0 3
[junit4:junit4] 2> 993863 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[x!23]} 0 6
[junit4:junit4] 2> 993865 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[y!24 (1437828911090106368)]} 0 0
[junit4:junit4] 2> 993872 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!24 (1437828911095349248)]} 0 0
[junit4:junit4] 2> 993873 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[y!24 (1437828911095349248)]} 0 3
[junit4:junit4] 2> 993874 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[y!24]} 0 6
[junit4:junit4] 2> 993876 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[z!25 (1437828911101640704)]} 0 0
[junit4:junit4] 2> 993884 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!25 (1437828911106883584)]} 0 0
[junit4:junit4] 2> 993885 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[z!25 (1437828911106883584)]} 0 4
[junit4:junit4] 2> 993886 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[z!25]} 0 7
[junit4:junit4] 2> 993888 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[a!26 (1437828911114223616)]} 0 0
[junit4:junit4] 2> 993895 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!26 (1437828911119466496)]} 0 0
[junit4:junit4] 2> 993896 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!26 (1437828911119466496)]} 0 3
[junit4:junit4] 2> 993897 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[a!26]} 0 6
[junit4:junit4] 2> 993899 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[b!27 (1437828911125757952)]} 0 0
[junit4:junit4] 2> 993905 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!27 (1437828911128903680)]} 0 0
[junit4:junit4] 2> 993906 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[b!27 (1437828911128903680)]} 0 4
[junit4:junit4] 2> 993909 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[c!28 (1437828911136243712)]} 0 0
[junit4:junit4] 2> 993917 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!28 (1437828911141486592)]} 0 0
[junit4:junit4] 2> 993918 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[c!28 (1437828911141486592)]} 0 4
[junit4:junit4] 2> 993919 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[c!28]} 0 7
[junit4:junit4] 2> 993921 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[d!29 (1437828911148826624)]} 0 0
[junit4:junit4] 2> 993927 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!29 (1437828911151972352)]} 0 0
[junit4:junit4] 2> 993928 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[d!29 (1437828911151972352)]} 0 4
[junit4:junit4] 2> 993931 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[e!30 (1437828911159312384)]} 0 0
[junit4:junit4] 2> 993938 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!30 (1437828911164555264)]} 0 0
[junit4:junit4] 2> 993939 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[e!30 (1437828911164555264)]} 0 3
[junit4:junit4] 2> 993940 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[e!30]} 0 6
[junit4:junit4] 2> 993942 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[f!31 (1437828911170846720)]} 0 0
[junit4:junit4] 2> 993950 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!31 (1437828911176089600)]} 0 0
[junit4:junit4] 2> 993951 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!31 (1437828911176089600)]} 0 4
[junit4:junit4] 2> 993952 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[f!31]} 0 7
[junit4:junit4] 2> 993954 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[g!32 (1437828911183429632)]} 0 0
[junit4:junit4] 2> 993960 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!32 (1437828911186575360)]} 0 0
[junit4:junit4] 2> 993961 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[g!32 (1437828911186575360)]} 0 4
[junit4:junit4] 2> 993964 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[h!33 (1437828911193915392)]} 0 0
[junit4:junit4] 2> 993972 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!33 (1437828911199158272)]} 0 0
[junit4:junit4] 2> 993973 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!33 (1437828911199158272)]} 0 4
[junit4:junit4] 2> 993974 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[h!33]} 0 7
[junit4:junit4] 2> 993976 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[i!34 (1437828911206498304)]} 0 0
[junit4:junit4] 2> 993982 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!34 (1437828911209644032)]} 0 0
[junit4:junit4] 2> 993983 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[i!34 (1437828911209644032)]} 0 4
[junit4:junit4] 2> 993986 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[j!35 (1437828911216984064)]} 0 0
[junit4:junit4] 2> 993991 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!35 (1437828911220129792)]} 0 0
[junit4:junit4] 2> 993992 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[j!35 (1437828911220129792)]} 0 3
[junit4:junit4] 2> 993995 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[k!36 (1437828911226421248)]} 0 0
[junit4:junit4] 2> 994002 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!36 (1437828911231664128)]} 0 0
[junit4:junit4] 2> 994003 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[k!36 (1437828911231664128)]} 0 3
[junit4:junit4] 2> 994004 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[k!36]} 0 6
[junit4:junit4] 2> 994006 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[l!37 (1437828911237955584)]} 0 0
[junit4:junit4] 2> 994014 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!37 (1437828911243198464)]} 0 0
[junit4:junit4] 2> 994015 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[l!37 (1437828911243198464)]} 0 4
[junit4:junit4] 2> 994016 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[l!37]} 0 7
[junit4:junit4] 2> 994018 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[m!38 (1437828911250538496)]} 0 0
[junit4:junit4] 2> 994024 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!38 (1437828911253684224)]} 0 0
[junit4:junit4] 2> 994025 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[m!38 (1437828911253684224)]} 0 4
[junit4:junit4] 2> 994028 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[n!39 (1437828911261024256)]} 0 0
[junit4:junit4] 2> 994034 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!39 (1437828911264169984)]} 0 0
[junit4:junit4] 2> 994035 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[n!39 (1437828911264169984)]} 0 4
[junit4:junit4] 2> 994038 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[o!40 (1437828911271510016)]} 0 0
[junit4:junit4] 2> 994046 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!40 (1437828911276752896)]} 0 0
[junit4:junit4] 2> 994047 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[o!40 (1437828911276752896)]} 0 4
[junit4:junit4] 2> 994048 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[o!40]} 0 7
[junit4:junit4] 2> 994050 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[p!41 (1437828911284092928)]} 0 0
[junit4:junit4] 2> 994058 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!41 (1437828911289335808)]} 0 0
[junit4:junit4] 2> 994059 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[p!41 (1437828911289335808)]} 0 4
[junit4:junit4] 2> 994060 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[p!41]} 0 7
[junit4:junit4] 2> 994062 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[q!42 (1437828911296675840)]} 0 0
[junit4:junit4] 2> 994068 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!42 (1437828911299821568)]} 0 0
[junit4:junit4] 2> 994069 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[q!42 (1437828911299821568)]} 0 4
[junit4:junit4] 2> 994072 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[r!43 (1437828911307161600)]} 0 0
[junit4:junit4] 2> 994078 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!43 (1437828911310307328)]} 0 0
[junit4:junit4] 2> 994079 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[r!43 (1437828911310307328)]} 0 4
[junit4:junit4] 2> 994082 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[s!44 (1437828911317647360)]} 0 0
[junit4:junit4] 2> 994090 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!44 (1437828911322890240)]} 0 0
[junit4:junit4] 2> 994091 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[s!44 (1437828911322890240)]} 0 4
[junit4:junit4] 2> 994092 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[s!44]} 0 7
[junit4:junit4] 2> 994094 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[t!45 (1437828911330230272)]} 0 0
[junit4:junit4] 2> 994100 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!45 (1437828911333376000)]} 0 0
[junit4:junit4] 2> 994101 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[t!45 (1437828911333376000)]} 0 4
[junit4:junit4] 2> 994104 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[u!46 (1437828911340716032)]} 0 0
[junit4:junit4] 2> 994108 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!46 (1437828911343861760)]} 0 0
[junit4:junit4] 2> 994109 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[u!46 (1437828911343861760)]} 0 2
[junit4:junit4] 2> 994112 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[v!47 (1437828911349104640)]} 0 0
[junit4:junit4] 2> 994119 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!47 (1437828911354347520)]} 0 0
[junit4:junit4] 2> 994120 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[v!47 (1437828911354347520)]} 0 3
[junit4:junit4] 2> 994121 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[v!47]} 0 6
[junit4:junit4] 2> 994123 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[w!48 (1437828911360638976)]} 0 0
[junit4:junit4] 2> 994131 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!48 (1437828911365881856)]} 0 0
[junit4:junit4] 2> 994132 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[w!48 (1437828911365881856)]} 0 4
[junit4:junit4] 2> 994133 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[w!48]} 0 7
[junit4:junit4] 2> 994135 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[x!49 (1437828911373221888)]} 0 0
[junit4:junit4] 2> 994141 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!49 (1437828911376367616)]} 0 0
[junit4:junit4] 2> 994142 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[x!49 (1437828911376367616)]} 0 4
[junit4:junit4] 2> 994145 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[y!50 (1437828911383707648)]} 0 0
[junit4:junit4] 2> 994150 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!50 (1437828911386853376)]} 0 0
[junit4:junit4] 2> 994151 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[y!50 (1437828911386853376)]} 0 3
[junit4:junit4] 2> 994154 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[z!51 (1437828911393144832)]} 0 0
[junit4:junit4] 2> 994159 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!51 (1437828911396290560)]} 0 0
[junit4:junit4] 2> 994160 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[z!51 (1437828911396290560)]} 0 3
[junit4:junit4] 2> 994163 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[a!52 (1437828911402582016)]} 0 0
[junit4:junit4] 2> 994171 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!52 (1437828911407824896)]} 0 0
[junit4:junit4] 2> 994172 T2856 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!52 (1437828911407824896)]} 0 4
[junit4:junit4] 2> 994173 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[a!52]} 0 7
[junit4:junit4] 2> 994175 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[b!53 (1437828911415164928)]} 0 0
[junit4:junit4] 2> 994183 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!53 (1437828911420407808)]} 0 0
[junit4:junit4] 2> 994184 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[b!53 (1437828911420407808)]} 0 4
[junit4:junit4] 2> 994185 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[b!53]} 0 7
[junit4:junit4] 2> 994187 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[c!54 (1437828911427747840)]} 0 0
[junit4:junit4] 2> 994195 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!54 (1437828911432990720)]} 0 0
[junit4:junit4] 2> 994196 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[c!54 (1437828911432990720)]} 0 4
[junit4:junit4] 2> 994197 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[c!54]} 0 7
[junit4:junit4] 2> 994199 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[d!55 (1437828911440330752)]} 0 0
[junit4:junit4] 2> 994205 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!55 (1437828911443476480)]} 0 0
[junit4:junit4] 2> 994206 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[d!55 (1437828911443476480)]} 0 4
[junit4:junit4] 2> 994209 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[e!56 (1437828911450816512)]} 0 0
[junit4:junit4] 2> 994215 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!56 (1437828911453962240)]} 0 0
[junit4:junit4] 2> 994216 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[e!56 (1437828911453962240)]} 0 4
[junit4:junit4] 2> 994219 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[f!57 (1437828911461302272)]} 0 0
[junit4:junit4] 2> 994224 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!57 (1437828911464448000)]} 0 0
[junit4:junit4] 2> 994225 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[f!57 (1437828911464448000)]} 0 3
[junit4:junit4] 2> 994228 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[g!58 (1437828911470739456)]} 0 0
[junit4:junit4] 2> 994234 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!58 (1437828911473885184)]} 0 0
[junit4:junit4] 2> 994235 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[g!58 (1437828911473885184)]} 0 4
[junit4:junit4] 2> 994238 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[h!59 (1437828911481225216)]} 0 0
[junit4:junit4] 2> 994246 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!59 (1437828911486468096)]} 0 0
[junit4:junit4] 2> 994247 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!59 (1437828911486468096)]} 0 4
[junit4:junit4] 2> 994248 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[h!59]} 0 7
[junit4:junit4] 2> 994250 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[i!60 (1437828911493808128)]} 0 0
[junit4:junit4] 2> 994257 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!60 (1437828911499051008)]} 0 0
[junit4:junit4] 2> 994258 T2838 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!60 (1437828911499051008)]} 0 3
[junit4:junit4] 2> 994259 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[i!60]} 0 6
[junit4:junit4] 2> 994261 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[j!61 (1437828911505342464)]} 0 0
[junit4:junit4] 2> 994267 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!61 (1437828911508488192)]} 0 0
[junit4:junit4] 2> 994268 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[j!61 (1437828911508488192)]} 0 4
[junit4:junit4] 2> 994271 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[k!62 (1437828911515828224)]} 0 0
[junit4:junit4] 2> 994277 T2868 C567 P56787 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56766/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!62 (1437828911518973952)]} 0 0
[junit4:junit4] 2> 994278 T2840 C571 P56766 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[k!62 (1437828911518973952)]} 0 4
[junit4:junit4] 2> 994281 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[l!63 (1437828911526313984)]} 0 0
[junit4:junit4] 2> 994287 T2883 C568 P56791 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={distrib.from=http://127.0.0.1:56772/yl_kf/f/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!63 (1437828911529459712)]} 0 0
[junit4:junit4] 2> 994288 T2854 C569 P56772 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[l!63 (1437828911529459712)]} 0 4
[junit4:junit4] 2> 994291 T2814 C570 P56762 oasup.LogUpdateProcessor.finish [collection1] webapp=/yl_kf/f path=/update params={wt=javabin&version=2} {add=[m!64 (1437828911536799744)]} 0 0
[junit4:junit4] 2> 994296 T2883 C568 P56791 oas
[...truncated too long message...]
8113 T2876 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1188113 T2876 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 1189185 T2893 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:56791/yl_kf/f couldn't connect to http://127.0.0.1:56772/yl_kf/f/collection1/, counting as success
[junit4:junit4] 2> 1189185 T2893 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56791/yl_kf/f DONE. sync succeeded
[junit4:junit4] 2> 1189186 T2893 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 1189186 T2893 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4:junit4] 2> 1189187 T2893 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7c576c0a
[junit4:junit4] 2> 1189194 T2893 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=689,adds=689,deletesById=331,deletesByQuery=0,errors=0,cumulative_adds=731,cumulative_deletesById=331,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 1189194 T2893 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1189195 T2893 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1189195 T2893 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1189208 T2893 C603 P56791 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46e4c69f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e99808e),segFN=segments_5,generation=5}
[junit4:junit4] 2> 1189209 T2893 C603 P56791 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 1189213 T2893 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1189213 T2893 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1189213 T2893 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index;done=false>>]
[junit4:junit4] 2> 1189214 T2893 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4/index
[junit4:junit4] 2> 1189214 T2893 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4;done=false>>]
[junit4:junit4] 2> 1189214 T2893 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371220474145/jetty4
[junit4:junit4] 2> 1189214 T2893 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1189215 T2893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1189215 T2893 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1189215 T2893 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {id=MockFixedIntBlock(blockSize=842), _version_=PostingsFormat(name=Asserting), n_ti=PostingsFormat(name=Asserting)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=hu_HU, timezone=Canada/Mountain
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=332562160,total=500170752
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ChaosMonkeyNothingIsSafeTest, DistributedSpellCheckComponentTest, SolrTestCaseJ4Test, TestReplicationHandler, ExternalFileFieldSortTest, ZkControllerTest, TestPHPSerializedResponseWriter, TestMaxScoreQueryParser, TestXIncludeConfig, ClusterStateUpdateTest, PreAnalyzedFieldTest, SliceStateTest, TestPerFieldSimilarity, JsonLoaderTest, CacheHeaderTest, TestRealTimeGet, TestFunctionQuery, TestCloudManagedSchema, TestLRUCache, SyncSliceTest, AutoCommitTest, OverseerCollectionProcessorTest, TestManagedSchema, TestWriterPerf, BadComponentTest, SynonymTokenizerTest, PeerSyncTest, UnloadDistributedZkTest, TestValueSourceCache, BinaryUpdateRequestHandlerTest, SOLR749Test, WordBreakSolrSpellCheckerTest, AliasIntegrationTest, SpellingQueryConverterTest, TestPropInjectDefaults, FileBasedSpellCheckerTest, FieldMutatingUpdateProcessorTest, TestFastLRUCache, TestDocumentBuilder, RecoveryZkTest, TestSerializedLuceneMatchVersion, SolrIndexSplitterTest, TestSearchPerf, SoftAutoCommitTest, TestSweetSpotSimilarityFactory, DocValuesTest, DirectSolrConnectionTest, TestCoreContainer, TestSolrQueryParser, TestFaceting, TestQuerySenderListener, CSVRequestHandlerTest, TestSolrDeletionPolicy1, MoreLikeThisHandlerTest, TestExtendedDismaxParser, TestDynamicFieldResource, TestPhraseSuggestions, TestLFUCache, TestStressVersions, TestFieldCollectionResource, TestReversedWildcardFilterFactory, TestSchemaSimilarityResource, ZkCLITest, BasicDistributedZkTest, DirectUpdateHandlerOptimizeTest, TestReloadAndDeleteDocs, TestSurroundQueryParser, DateFieldTest, RAMDirectoryFactoryTest, SystemInfoHandlerTest, TestPseudoReturnFields, ShardSplitTest]
[junit4:junit4] Completed on J0 in 225.60s, 1 test, 1 error <<< FAILURES!
[...truncated 460 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:385: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:365: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 296 suites, 1243 tests, 1 error, 13 ignored (7 assumptions)
Total time: 52 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure