You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/24 22:59:36 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6293 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6293/
Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<704> but was:<58>
Stack Trace:
java.lang.AssertionError: expected:<704> but was:<58>
at __randomizedtesting.SeedInfo.seed([A1C5ACADBDB94B8E:202322B5CAE62BB2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9589 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 557456 T1754 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /z/r
[junit4:junit4] 2> 557492 T1754 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1372107056390
[junit4:junit4] 2> 557496 T1754 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 557498 T1755 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 557597 T1754 oasc.ZkTestServer.run start zk server on port:50294
[junit4:junit4] 2> 557601 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 557898 T1761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@424496 name:ZooKeeperConnection Watcher:127.0.0.1:50294 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 557899 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 557900 T1754 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 557908 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 557913 T1763 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@85b82e name:ZooKeeperConnection Watcher:127.0.0.1:50294/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 557913 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 557914 T1754 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 557919 T1754 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 557922 T1754 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 557925 T1754 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 557929 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 557931 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 557938 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 557940 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 557950 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 557952 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 557955 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 557956 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 557959 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 557960 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 557964 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 557966 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 557970 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 557971 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 557976 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 557976 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 557980 T1754 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 557980 T1754 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 558183 T1754 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 558296 T1754 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59100
[junit4:junit4] 2> 558298 T1754 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 558299 T1754 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 558299 T1754 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903
[junit4:junit4] 2> 558300 T1754 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/solr.xml
[junit4:junit4] 2> 558300 T1754 oasc.CoreContainer.<init> New CoreContainer 27491297
[junit4:junit4] 2> 558301 T1754 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/'
[junit4:junit4] 2> 558302 T1754 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/'
[junit4:junit4] 2> 558669 T1754 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 558670 T1754 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 558670 T1754 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 558671 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 558671 T1754 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 558672 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 558672 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 558673 T1754 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 558674 T1754 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 558675 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 558976 T1754 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 558976 T1754 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50294/solr
[junit4:junit4] 2> 558977 T1754 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 558979 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 558983 T1774 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d284c name:ZooKeeperConnection Watcher:127.0.0.1:50294 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 558983 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 558985 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 558997 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 558999 T1776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18e65ee name:ZooKeeperConnection Watcher:127.0.0.1:50294/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 559000 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 559002 T1754 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 559005 T1754 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 559009 T1754 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 559011 T1754 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59100_z%2Fr
[junit4:junit4] 2> 559012 T1754 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59100_z%2Fr
[junit4:junit4] 2> 559015 T1754 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 559019 T1754 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 559021 T1754 oasc.Overseer.start Overseer (id=89922408048558083-127.0.0.1:59100_z%2Fr-n_0000000000) starting
[junit4:junit4] 2> 559025 T1754 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 559028 T1754 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 559028 T1778 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 559032 T1754 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 559034 T1754 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 559036 T1777 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 559053 T1779 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 559054 T1779 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 560540 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 560541 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59100/z/r",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:59100_z%2Fr",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 560542 T1777 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 560544 T1777 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 560548 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 561057 T1779 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/collection1
[junit4:junit4] 2> 561057 T1779 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 561059 T1779 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 561060 T1779 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 561062 T1779 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/collection1/'
[junit4:junit4] 2> 561065 T1779 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/collection1/lib/README' to classloader
[junit4:junit4] 2> 561067 T1779 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 561246 T1779 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 561516 T1779 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 561521 T1779 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 561533 T1779 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 562428 T1779 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 562437 T1779 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 562440 T1779 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 562456 T1779 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 562458 T1779 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 562461 T1779 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 562462 T1779 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 562462 T1779 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 562462 T1779 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 562463 T1779 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 562463 T1779 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 562463 T1779 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 562464 T1779 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1372107056903/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/control/data/
[junit4:junit4] 2> 562464 T1779 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11f1749
[junit4:junit4] 2> 562465 T1779 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 562465 T1779 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/control/data
[junit4:junit4] 2> 562465 T1779 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/control/data/index/
[junit4:junit4] 2> 562466 T1779 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1372107056389/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 562466 T1779 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/control/data/index
[junit4:junit4] 2> 562469 T1779 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3bcaf0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@152dc3f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 562469 T1779 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 562472 T1779 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 562472 T1779 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 562473 T1779 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 562473 T1779 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 562474 T1779 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 562474 T1779 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 562474 T1779 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 562474 T1779 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 562475 T1779 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 562478 T1779 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 562480 T1779 oass.SolrIndexSearcher.<init> Opening Searcher@10fa6d5 main
[junit4:junit4] 2> 562481 T1779 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/control/data/tlog
[junit4:junit4] 2> 562481 T1779 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 562482 T1779 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 562486 T1780 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10fa6d5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 562487 T1779 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 562487 T1779 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59100/z/r collection:control_collection shard:shard1
[junit4:junit4] 2> 562488 T1779 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 562492 T1779 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 562494 T1779 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 562494 T1779 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 562494 T1779 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59100/z/r/collection1/
[junit4:junit4] 2> 562495 T1779 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 562495 T1779 oasc.SyncStrategy.syncToMe http://127.0.0.1:59100/z/r/collection1/ has no replicas
[junit4:junit4] 2> 562495 T1779 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59100/z/r/collection1/
[junit4:junit4] 2> 562495 T1779 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 563552 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 563559 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 563600 T1779 oasc.ZkController.register We are http://127.0.0.1:59100/z/r/collection1/ and leader is http://127.0.0.1:59100/z/r/collection1/
[junit4:junit4] 2> 563600 T1779 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59100/z/r
[junit4:junit4] 2> 563601 T1779 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 563601 T1779 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 563601 T1779 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 563603 T1779 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 563604 T1754 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 563604 T1754 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 563605 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 563607 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 563609 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 563618 T1783 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1586318 name:ZooKeeperConnection Watcher:127.0.0.1:50294/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 563618 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 563619 T1754 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 563621 T1754 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 563685 T1754 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 563689 T1754 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42558
[junit4:junit4] 2> 563690 T1754 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 563691 T1754 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 563691 T1754 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518
[junit4:junit4] 2> 563691 T1754 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/solr.xml
[junit4:junit4] 2> 563691 T1754 oasc.CoreContainer.<init> New CoreContainer 30105815
[junit4:junit4] 2> 563692 T1754 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/'
[junit4:junit4] 2> 563692 T1754 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/'
[junit4:junit4] 2> 563745 T1754 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 563745 T1754 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 563745 T1754 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 563746 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 563746 T1754 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 563746 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 563746 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 563747 T1754 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 563747 T1754 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 563747 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 563750 T1754 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 563750 T1754 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50294/solr
[junit4:junit4] 2> 563751 T1754 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 563752 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 563753 T1794 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42b020 name:ZooKeeperConnection Watcher:127.0.0.1:50294 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 563754 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 563755 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 563759 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 563760 T1796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18cc69d name:ZooKeeperConnection Watcher:127.0.0.1:50294/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 563760 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 563763 T1754 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 564765 T1754 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42558_z%2Fr
[junit4:junit4] 2> 564767 T1754 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42558_z%2Fr
[junit4:junit4] 2> 564775 T1776 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 564775 T1796 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 564775 T1783 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 564775 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 564781 T1797 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 564781 T1797 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 565062 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 565063 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59100/z/r",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:59100_z%2Fr",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 565065 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42558/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:42558_z%2Fr",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 565065 T1777 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 565065 T1777 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 565068 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 565068 T1783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 565068 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 565782 T1797 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/collection1
[junit4:junit4] 2> 565782 T1797 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 565783 T1797 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 565783 T1797 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 565784 T1797 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/collection1/'
[junit4:junit4] 2> 565785 T1797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/collection1/lib/README' to classloader
[junit4:junit4] 2> 565785 T1797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 565814 T1797 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 565844 T1797 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 565845 T1797 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 565848 T1797 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 566194 T1797 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 566199 T1797 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 566202 T1797 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 566211 T1797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 566214 T1797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 566216 T1797 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 566217 T1797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 566218 T1797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 566218 T1797 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 566219 T1797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 566219 T1797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 566220 T1797 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 566220 T1797 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1372107062518/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty1/
[junit4:junit4] 2> 566220 T1797 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11f1749
[junit4:junit4] 2> 566220 T1797 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 566221 T1797 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty1
[junit4:junit4] 2> 566221 T1797 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty1/index/
[junit4:junit4] 2> 566221 T1797 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 566221 T1797 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty1/index
[junit4:junit4] 2> 566223 T1797 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60049b lockFactory=org.apache.lucene.store.NativeFSLockFactory@114890b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 566223 T1797 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 566224 T1797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 566225 T1797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 566225 T1797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 566225 T1797 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 566226 T1797 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 566226 T1797 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 566226 T1797 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 566227 T1797 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 566227 T1797 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 566228 T1797 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 566230 T1797 oass.SolrIndexSearcher.<init> Opening Searcher@875a67 main
[junit4:junit4] 2> 566231 T1797 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty1/tlog
[junit4:junit4] 2> 566231 T1797 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 566231 T1797 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 566235 T1798 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@875a67 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 566236 T1797 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 566236 T1797 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42558/z/r collection:collection1 shard:shard1
[junit4:junit4] 2> 566237 T1797 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 566241 T1797 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 566242 T1797 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 566243 T1797 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 566243 T1797 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42558/z/r/collection1/
[junit4:junit4] 2> 566243 T1797 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 566243 T1797 oasc.SyncStrategy.syncToMe http://127.0.0.1:42558/z/r/collection1/ has no replicas
[junit4:junit4] 2> 566243 T1797 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42558/z/r/collection1/
[junit4:junit4] 2> 566243 T1797 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 566572 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 566578 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 566578 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 566578 T1783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 566597 T1797 oasc.ZkController.register We are http://127.0.0.1:42558/z/r/collection1/ and leader is http://127.0.0.1:42558/z/r/collection1/
[junit4:junit4] 2> 566598 T1797 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42558/z/r
[junit4:junit4] 2> 566598 T1797 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 566598 T1797 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 566598 T1797 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 566599 T1797 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 566600 T1754 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 566600 T1754 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 566601 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 566667 T1754 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 566673 T1754 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47680
[junit4:junit4] 2> 566673 T1754 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 566674 T1754 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 566674 T1754 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499
[junit4:junit4] 2> 566675 T1754 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/solr.xml
[junit4:junit4] 2> 566675 T1754 oasc.CoreContainer.<init> New CoreContainer 13725951
[junit4:junit4] 2> 566675 T1754 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/'
[junit4:junit4] 2> 566676 T1754 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/'
[junit4:junit4] 2> 566724 T1754 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 566725 T1754 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 566725 T1754 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 566725 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 566726 T1754 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 566726 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 566726 T1754 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 566726 T1754 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 566727 T1754 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 566727 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 566731 T1754 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 566731 T1754 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50294/solr
[junit4:junit4] 2> 566732 T1754 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 566733 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 566735 T1810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e2e13 name:ZooKeeperConnection Watcher:127.0.0.1:50294 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 566736 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 566738 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 566740 T1754 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 566817 T1812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@310988 name:ZooKeeperConnection Watcher:127.0.0.1:50294/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 566818 T1754 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 566825 T1754 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 567827 T1754 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47680_z%2Fr
[junit4:junit4] 2> 567829 T1754 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47680_z%2Fr
[junit4:junit4] 2> 567835 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 567835 T1783 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 567835 T1776 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 567836 T1783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 567836 T1812 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 567836 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 567837 T1796 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 567841 T1813 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 567841 T1813 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 568082 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 568082 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42558/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:42558_z%2Fr",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 568085 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47680/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:47680_z%2Fr",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 568085 T1777 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 568085 T1777 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 568088 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 568088 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 568088 T1783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 568088 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 568842 T1813 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/collection1
[junit4:junit4] 2> 568843 T1813 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 568843 T1813 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 568843 T1813 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 568844 T1813 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/collection1/'
[junit4:junit4] 2> 568844 T1813 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/collection1/lib/README' to classloader
[junit4:junit4] 2> 568845 T1813 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 568876 T1813 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 568904 T1813 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 568905 T1813 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 568908 T1813 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 569217 T1813 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 569222 T1813 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 569224 T1813 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 569233 T1813 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 569236 T1813 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 569238 T1813 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 569238 T1813 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 569239 T1813 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 569239 T1813 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 569240 T1813 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 569240 T1813 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 569240 T1813 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 569240 T1813 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1372107065499/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/
[junit4:junit4] 2> 569240 T1813 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11f1749
[junit4:junit4] 2> 569241 T1813 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 569241 T1813 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2
[junit4:junit4] 2> 569242 T1813 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index/
[junit4:junit4] 2> 569242 T1813 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 569242 T1813 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index
[junit4:junit4] 2> 569243 T1813 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_1,generation=1}
[junit4:junit4] 2> 569243 T1813 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 569245 T1813 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 569245 T1813 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 569246 T1813 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 569246 T1813 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 569247 T1813 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 569247 T1813 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 569247 T1813 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 569247 T1813 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 569248 T1813 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 569249 T1813 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 569251 T1813 oass.SolrIndexSearcher.<init> Opening Searcher@463ced main
[junit4:junit4] 2> 569251 T1813 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/tlog
[junit4:junit4] 2> 569252 T1813 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 569252 T1813 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 569255 T1814 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@463ced main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 569257 T1813 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 569257 T1813 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47680/z/r collection:collection1 shard:shard1
[junit4:junit4] 2> 569259 T1813 oasc.ZkController.register We are http://127.0.0.1:47680/z/r/collection1/ and leader is http://127.0.0.1:42558/z/r/collection1/
[junit4:junit4] 2> 569260 T1813 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47680/z/r
[junit4:junit4] 2> 569260 T1813 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 569260 T1813 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C275 name=collection1 org.apache.solr.core.SolrCore@146d4d6 url=http://127.0.0.1:47680/z/r/collection1 node=127.0.0.1:47680_z%2Fr C275_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:47680/z/r, collection=collection1, state=down, node_name=127.0.0.1:47680_z%2Fr, core=collection1}
[junit4:junit4] 2> 569261 T1815 C275 P47680 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 569261 T1815 C275 P47680 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 569262 T1815 C275 P47680 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 569262 T1813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 569262 T1815 C275 P47680 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 569263 T1815 C275 P47680 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 569263 T1754 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 569263 T1754 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 569264 T1754 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 569267 T1754 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 569282 T1791 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C276 name=collection1 org.apache.solr.core.SolrCore@138a48f url=http://127.0.0.1:59100/z/r/collection1 node=127.0.0.1:59100_z%2Fr C276_STATE=coll:control_collection core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:59100/z/r, collection=control_collection, state=active, node_name=127.0.0.1:59100_z%2Fr, core=collection1, leader=true}
[junit4:junit4] 2> 569288 T1771 C276 P59100 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3bcaf0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@152dc3f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 569288 T1771 C276 P59100 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 569296 T1768 C276 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10001 (1438758541126008833)]} 0 11
[junit4:junit4] 2> 569296 T1771 C276 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[1 (1438758541126008832)]} 0 11
[junit4:junit4] 2> ASYNC NEW_CORE C277 name=collection1 org.apache.solr.core.SolrCore@11e2806 url=http://127.0.0.1:42558/z/r/collection1 node=127.0.0.1:42558_z%2Fr C277_STATE=coll:collection1 core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:42558/z/r, collection=collection1, state=active, node_name=127.0.0.1:42558_z%2Fr, core=collection1, leader=true}
[junit4:junit4] 2> 569308 T1788 C277 P42558 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60049b lockFactory=org.apache.lucene.store.NativeFSLockFactory@114890b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 569309 T1788 C277 P42558 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 569312 T1792 C277 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10001 (1438758541148028929)]} 0 5
[junit4:junit4] 2> 569312 T1788 C277 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[1 (1438758541148028928)]} 0 5
[junit4:junit4] 2> 569420 T1770 C276 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[1 (-1438758541265469440)]} 0 1
[junit4:junit4] 2> 569423 T1772 C276 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10001 (-1438758541266518016)]} 0 4
[junit4:junit4] 2> ASYNC NEW_CORE C278 name=collection1 org.apache.solr.core.SolrCore@11e2806 url=http://127.0.0.1:42558/z/r/collection1 node=127.0.0.1:42558_z%2Fr C278_STATE=coll:collection1 core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:42558/z/r, collection=collection1, state=active, node_name=127.0.0.1:42558_z%2Fr, core=collection1, leader=true}
[junit4:junit4] 2> 569443 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[1 (-1438758541289586688)]} 0 1
[junit4:junit4] 2> 569443 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10001 (-1438758541289586689)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C279 name=collection1 org.apache.solr.core.SolrCore@138a48f url=http://127.0.0.1:59100/z/r/collection1 node=127.0.0.1:59100_z%2Fr C279_STATE=coll:control_collection core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:59100/z/r, collection=control_collection, state=active, node_name=127.0.0.1:59100_z%2Fr, core=collection1, leader=true}
[junit4:junit4] 2> 569446 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[2 (1438758541292732416)]} 0 1
[junit4:junit4] 2> 569446 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10002 (1438758541293780992)]} 0 1
[junit4:junit4] 2> 569449 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[2 (1438758541296926720)]} 0 1
[junit4:junit4] 2> 569449 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10002 (1438758541296926721)]} 0 0
[junit4:junit4] 2> 569467 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[3 (1438758541314752512)]} 0 1
[junit4:junit4] 2> 569467 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10003 (1438758541314752513)]} 0 1
[junit4:junit4] 2> 569469 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[3 (1438758541317898240)]} 0 0
[junit4:junit4] 2> 569470 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10003 (1438758541317898241)]} 0 1
[junit4:junit4] 2> 569482 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[2 (-1438758541330481152)]} 0 1
[junit4:junit4] 2> 569482 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10002 (-1438758541331529728)]} 0 0
[junit4:junit4] 2> 569484 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[2 (-1438758541333626880)]} 0 0
[junit4:junit4] 2> 569484 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10002 (-1438758541333626881)]} 0 0
[junit4:junit4] 2> 569486 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[4 (1438758541335724032)]} 0 1
[junit4:junit4] 2> 569486 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10004 (1438758541335724033)]} 0 0
[junit4:junit4] 2> 569488 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[4 (1438758541337821184)]} 0 0
[junit4:junit4] 2> 569489 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10004 (1438758541337821185)]} 0 1
[junit4:junit4] 2> 569526 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[5 (1438758541377667072)]} 0 1
[junit4:junit4] 2> 569526 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10005 (1438758541377667073)]} 0 0
[junit4:junit4] 2> 569529 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[5 (1438758541379764224)]} 0 1
[junit4:junit4] 2> 569529 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10005 (1438758541379764225)]} 0 1
[junit4:junit4] 2> 569592 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 569593 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47680/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "node_name":"127.0.0.1:47680_z%2Fr",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 569596 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 569596 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 569596 T1783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 569596 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 569613 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10003 (-1438758541468893185)]} 0 0
[junit4:junit4] 2> 569613 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[3 (-1438758541468893184)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C280 name=collection1 org.apache.solr.core.SolrCore@146d4d6 url=http://127.0.0.1:47680/z/r/collection1 node=127.0.0.1:47680_z%2Fr C280_STATE=coll:collection1 core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:47680/z/r, collection=collection1, state=recovering, node_name=127.0.0.1:47680_z%2Fr, core=collection1}
[junit4:junit4] 2> 569622 T1808 C280 P47680 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_1,generation=1}
[junit4:junit4] 2> 569623 T1808 C280 P47680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 569624 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[3 (-1438758541470990337)]} 0 3
[junit4:junit4] 2> 569624 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10003 (-1438758541470990336)]} 0 3
[junit4:junit4] 2> 569624 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[3 (-1438758541470990337)]} 0 9
[junit4:junit4] 2> 569624 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10003 (-1438758541470990336)]} 0 9
[junit4:junit4] 2> 569627 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[6 (1438758541483573248)]} 0 1
[junit4:junit4] 2> 569627 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10006 (1438758541483573249)]} 0 0
[junit4:junit4] 2> 569633 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[6 (1438758541485670400)]} 0 1
[junit4:junit4] 2> 569633 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10006 (1438758541486718976)]} 0 1
[junit4:junit4] 2> 569633 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[6 (1438758541485670400)]} 0 4
[junit4:junit4] 2> 569634 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10006 (1438758541486718976)]} 0 4
[junit4:junit4] 2> 569637 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[4 (-1438758541494059008)]} 0 0
[junit4:junit4] 2> 569637 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10004 (-1438758541494059009)]} 0 0
[junit4:junit4] 2> 569642 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[4 (-1438758541496156160)]} 0 1
[junit4:junit4] 2> 569642 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[4 (-1438758541496156160)]} 0 3
[junit4:junit4] 2> 569642 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10004 (-1438758541497204736)]} 0 0
[junit4:junit4] 2> 569643 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10004 (-1438758541497204736)]} 0 3
[junit4:junit4] 2> 569645 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[7 (1438758541501399040)]} 0 1
[junit4:junit4] 2> 569647 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10007 (1438758541503496192)]} 0 1
[junit4:junit4] 2> 569651 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[7 (1438758541504544768)]} 0 1
[junit4:junit4] 2> 569652 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[7 (1438758541504544768)]} 0 5
[junit4:junit4] 2> 569654 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10007 (1438758541507690496)]} 0 1
[junit4:junit4] 2> 569655 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10007 (1438758541507690496)]} 0 5
[junit4:junit4] 2> 569665 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[6 (-1438758541523419136)]} 0 0
[junit4:junit4] 2> 569667 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10006 (-1438758541525516288)]} 0 0
[junit4:junit4] 2> 569669 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[6 (-1438758541525516288)]} 0 0
[junit4:junit4] 2> 569670 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[6 (-1438758541525516288)]} 0 2
[junit4:junit4] 2> 569670 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10006 (-1438758541527613440)]} 0 0
[junit4:junit4] 2> 569671 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10006 (-1438758541527613440)]} 0 2
[junit4:junit4] 2> 569672 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[8 (1438758541529710592)]} 0 1
[junit4:junit4] 2> 569673 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10008 (1438758541531807744)]} 0 0
[junit4:junit4] 2> 569676 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[8 (1438758541532856320)]} 0 0
[junit4:junit4] 2> 569677 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[8 (1438758541532856320)]} 0 3
[junit4:junit4] 2> 569678 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10008 (1438758541534953472)]} 0 0
[junit4:junit4] 2> 569678 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10008 (1438758541534953472)]} 0 3
[junit4:junit4] 2> 569726 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[9 (1438758541586333696)]} 0 1
[junit4:junit4] 2> 569727 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10009 (1438758541587382272)]} 0 1
[junit4:junit4] 2> 569731 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[9 (1438758541589479424)]} 0 1
[junit4:junit4] 2> 569731 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[9 (1438758541589479424)]} 0 3
[junit4:junit4] 2> 569732 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10009 (1438758541590528000)]} 0 1
[junit4:junit4] 2> 569732 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10009 (1438758541590528000)]} 0 3
[junit4:junit4] 2> 569763 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[9 (-1438758541626179584)]} 0 0
[junit4:junit4] 2> 569764 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10009 (-1438758541626179585)]} 0 1
[junit4:junit4] 2> 569768 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[9 (-1438758541629325312)]} 0 1
[junit4:junit4] 2> 569768 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10009 (-1438758541629325313)]} 0 0
[junit4:junit4] 2> 569768 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[9 (-1438758541629325312)]} 0 2
[junit4:junit4] 2> 569768 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10009 (-1438758541629325313)]} 0 2
[junit4:junit4] 2> 569771 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10 (1438758541633519616)]} 0 1
[junit4:junit4] 2> 569771 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10010 (1438758541633519617)]} 0 1
[junit4:junit4] 2> 569775 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10 (1438758541635616768)]} 0 1
[junit4:junit4] 2> 569776 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10010 (1438758541636665344)]} 0 0
[junit4:junit4] 2> 569776 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10 (1438758541635616768)]} 0 4
[junit4:junit4] 2> 569776 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10010 (1438758541636665344)]} 0 3
[junit4:junit4] 2> 569820 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10011 (1438758541684899840)]} 0 1
[junit4:junit4] 2> 569820 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[11 (1438758541684899841)]} 0 1
[junit4:junit4] 2> 569825 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10011 (1438758541688045568)]} 0 1
[junit4:junit4] 2> 569825 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[11 (1438758541688045569)]} 0 1
[junit4:junit4] 2> 569825 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10011 (1438758541688045568)]} 0 3
[junit4:junit4] 2> 569825 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[11 (1438758541688045569)]} 0 3
[junit4:junit4] 2> 569926 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[11 (-1438758541797097473)]} 0 1
[junit4:junit4] 2> 569926 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10011 (-1438758541797097472)]} 0 1
[junit4:junit4] 2> 569930 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10011 (-1438758541799194625)]} 0 0
[junit4:junit4] 2> 569930 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[11 (-1438758541799194624)]} 0 0
[junit4:junit4] 2> 569931 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10011 (-1438758541799194625)]} 0 3
[junit4:junit4] 2> 569931 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[11 (-1438758541799194624)]} 0 3
[junit4:junit4] 2> 569933 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10012 (1438758541804437504)]} 0 0
[junit4:junit4] 2> 569934 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[12 (1438758541804437505)]} 0 1
[junit4:junit4] 2> 569938 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10012 (1438758541806534656)]} 0 1
[junit4:junit4] 2> 569938 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[12 (1438758541807583232)]} 0 0
[junit4:junit4] 2> 569938 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10012 (1438758541806534656)]} 0 3
[junit4:junit4] 2> 569939 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[12 (1438758541807583232)]} 0 3
[junit4:junit4] 2> 570022 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[13 (1438758541897760768)]} 0 0
[junit4:junit4] 2> 570022 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10013 (1438758541897760769)]} 0 0
[junit4:junit4] 2> 570027 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[13 (1438758541900906496)]} 0 0
[junit4:junit4] 2> 570028 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10013 (1438758541900906497)]} 0 1
[junit4:junit4] 2> 570028 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[13 (1438758541900906496)]} 0 3
[junit4:junit4] 2> 570028 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10013 (1438758541900906497)]} 0 3
[junit4:junit4] 2> 570105 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10014 (1438758541984792576)]} 0 1
[junit4:junit4] 2> 570106 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[14 (1438758541984792577)]} 0 2
[junit4:junit4] 2> 570112 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10014 (1438758541987938304)]} 0 1
[junit4:junit4] 2> 570113 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10014 (1438758541987938304)]} 0 5
[junit4:junit4] 2> 570113 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[14 (1438758541988986880)]} 0 1
[junit4:junit4] 2> 570115 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[14 (1438758541988986880)]} 0 6
[junit4:junit4] 2> 570127 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10014 (-1438758542007861248)]} 0 0
[junit4:junit4] 2> 570129 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[14 (-1438758542009958400)]} 0 0
[junit4:junit4] 2> 570132 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10014 (-1438758542011006976)]} 0 1
[junit4:junit4] 2> 570132 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10014 (-1438758542011006976)]} 0 2
[junit4:junit4] 2> 570133 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[14 (-1438758542012055552)]} 0 0
[junit4:junit4] 2> 570133 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[14 (-1438758542012055552)]} 0 2
[junit4:junit4] 2> 570134 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10015 (1438758542015201280)]} 0 0
[junit4:junit4] 2> 570135 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[15 (1438758542016249856)]} 0 0
[junit4:junit4] 2> 570139 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10015 (1438758542017298432)]} 0 1
[junit4:junit4] 2> 570139 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10015 (1438758542017298432)]} 0 3
[junit4:junit4] 2> 570140 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[15 (1438758542018347008)]} 0 1
[junit4:junit4] 2> 570141 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[15 (1438758542018347008)]} 0 4
[junit4:junit4] 2> 570196 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10016 (1438758542080212992)]} 0 1
[junit4:junit4] 2> 570197 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[16 (1438758542081261568)]} 0 1
[junit4:junit4] 2> 570201 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10016 (1438758542082310144)]} 0 1
[junit4:junit4] 2> 570201 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10016 (1438758542082310144)]} 0 3
[junit4:junit4] 2> 570201 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[16 (1438758542083358720)]} 0 0
[junit4:junit4] 2> 570202 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[16 (1438758542083358720)]} 0 3
[junit4:junit4] 2> 570255 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10017 (1438758542141030400)]} 0 1
[junit4:junit4] 2> 570255 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[17 (1438758542142078976)]} 0 1
[junit4:junit4] 2> 570263 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10017 (1438758542145224704)]} 0 1
[junit4:junit4] 2> 570264 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[17 (1438758542146273280)]} 0 2
[junit4:junit4] 2> 570264 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10017 (1438758542145224704)]} 0 6
[junit4:junit4] 2> 570265 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[17 (1438758542146273280)]} 0 6
[junit4:junit4] 2> 570283 T1791 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 570284 T1791 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:47680_z%252Fr&wt=javabin&state=recovering&onlyIfLeader=true&coreNodeName=3&checkLive=true&core=collection1&version=2&action=PREPRECOVERY} status=0 QTime=1002
[junit4:junit4] 2> 570357 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[18 (1438758542247985153)]} 0 1
[junit4:junit4] 2> 570357 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10018 (1438758542247985152)]} 0 2
[junit4:junit4] 2> 570364 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[18 (1438758542252179456)]} 0 1
[junit4:junit4] 2> 570365 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[18 (1438758542252179456)]} 0 5
[junit4:junit4] 2> 570365 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10018 (1438758542252179457)]} 0 1
[junit4:junit4] 2> 570366 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10018 (1438758542252179457)]} 0 6
[junit4:junit4] 2> 570448 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[19 (1438758542343405568)]} 0 1
[junit4:junit4] 2> 570448 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10019 (1438758542344454144)]} 0 0
[junit4:junit4] 2> 570453 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[19 (1438758542346551296)]} 0 0
[junit4:junit4] 2> 570454 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[19 (1438758542346551296)]} 0 4
[junit4:junit4] 2> 570454 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10019 (1438758542347599872)]} 0 1
[junit4:junit4] 2> 570454 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10019 (1438758542347599872)]} 0 3
[junit4:junit4] 2> 570468 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[16 (-1438758542365425664)]} 0 0
[junit4:junit4] 2> 570469 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10016 (-1438758542366474240)]} 0 1
[junit4:junit4] 2> 570472 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10016 (-1438758542368571393)]} 0 0
[junit4:junit4] 2> 570472 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[16 (-1438758542368571392)]} 0 0
[junit4:junit4] 2> 570473 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10016 (-1438758542368571393)]} 0 2
[junit4:junit4] 2> 570473 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[16 (-1438758542368571392)]} 0 3
[junit4:junit4] 2> 570476 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10020 (1438758542372765696)]} 0 1
[junit4:junit4] 2> 570476 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[20 (1438758542373814272)]} 0 0
[junit4:junit4] 2> 570482 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[20 (1438758542376960001)]} 0 1
[junit4:junit4] 2> 570482 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10020 (1438758542376960000)]} 0 1
[junit4:junit4] 2> 570482 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[20 (1438758542376960001)]} 0 4
[junit4:junit4] 2> 570482 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10020 (1438758542376960000)]} 0 4
[junit4:junit4] 2> 570486 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[21 (1438758542383251456)]} 0 1
[junit4:junit4] 2> 570486 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10021 (1438758542384300032)]} 0 1
[junit4:junit4] 2> 570495 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[21 (1438758542386397184)]} 0 1
[junit4:junit4] 2> 570495 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10021 (1438758542387445760)]} 0 1
[junit4:junit4] 2> 570495 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[21 (1438758542386397184)]} 0 7
[junit4:junit4] 2> 570496 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10021 (1438758542387445760)]} 0 7
[junit4:junit4] 2> 570529 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10019 (-1438758542429388800)]} 0 0
[junit4:junit4] 2> 570529 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[19 (-1438758542429388801)]} 0 0
[junit4:junit4] 2> 570533 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[19 (-1438758542431485953)]} 0 0
[junit4:junit4] 2> 570533 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10019 (-1438758542431485952)]} 0 0
[junit4:junit4] 2> 570534 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[19 (-1438758542431485953)]} 0 3
[junit4:junit4] 2> 570534 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10019 (-1438758542431485952)]} 0 3
[junit4:junit4] 2> 570536 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[22 (1438758542436728832)]} 0 0
[junit4:junit4] 2> 570536 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10022 (1438758542436728833)]} 0 0
[junit4:junit4] 2> 570540 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[22 (1438758542438825984)]} 0 0
[junit4:junit4] 2> 570541 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[22 (1438758542438825984)]} 0 3
[junit4:junit4] 2> 570541 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10022 (1438758542438825985)]} 0 1
[junit4:junit4] 2> 570542 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10022 (1438758542438825985)]} 0 4
[junit4:junit4] 2> 570599 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[23 (1438758542502789121)]} 0 0
[junit4:junit4] 2> 570600 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10023 (1438758542502789120)]} 0 1
[junit4:junit4] 2> 570604 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[23 (1438758542505934848)]} 0 0
[junit4:junit4] 2> 570605 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10023 (1438758542505934849)]} 0 1
[junit4:junit4] 2> 570605 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[23 (1438758542505934848)]} 0 3
[junit4:junit4] 2> 570605 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10023 (1438758542505934849)]} 0 3
[junit4:junit4] 2> 570671 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10020 (-1438758542578286593)]} 0 1
[junit4:junit4] 2> 570671 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[20 (-1438758542578286592)]} 0 1
[junit4:junit4] 2> 570675 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[20 (-1438758542580383745)]} 0 0
[junit4:junit4] 2> 570675 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10020 (-1438758542580383744)]} 0 0
[junit4:junit4] 2> 570675 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[20 (-1438758542580383745)]} 0 2
[junit4:junit4] 2> 570676 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10020 (-1438758542580383744)]} 0 3
[junit4:junit4] 2> 570678 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[24 (1438758542584578048)]} 0 1
[junit4:junit4] 2> 570678 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10024 (1438758542585626624)]} 0 0
[junit4:junit4] 2> 570683 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[24 (1438758542587723776)]} 0 1
[junit4:junit4] 2> 570683 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10024 (1438758542587723777)]} 0 0
[junit4:junit4] 2> 570683 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[24 (1438758542587723776)]} 0 3
[junit4:junit4] 2> 570684 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10024 (1438758542587723777)]} 0 4
[junit4:junit4] 2> 570767 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[25 (1438758542677901312)]} 0 1
[junit4:junit4] 2> 570767 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10025 (1438758542677901313)]} 0 1
[junit4:junit4] 2> 570771 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[25 (1438758542681047040)]} 0 0
[junit4:junit4] 2> 570772 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10025 (1438758542681047041)]} 0 1
[junit4:junit4] 2> 570772 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[25 (1438758542681047040)]} 0 3
[junit4:junit4] 2> 570772 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10025 (1438758542681047041)]} 0 3
[junit4:junit4] 2> 570807 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[21 (-1438758542720892928)]} 0 0
[junit4:junit4] 2> 570807 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10021 (-1438758542720892929)]} 0 0
[junit4:junit4] 2> 570811 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10021 (-1438758542722990081)]} 0 0
[junit4:junit4] 2> 570811 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[21 (-1438758542722990080)]} 0 0
[junit4:junit4] 2> 570812 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10021 (-1438758542722990081)]} 0 3
[junit4:junit4] 2> 570812 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[21 (-1438758542722990080)]} 0 3
[junit4:junit4] 2> 570815 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[26 (1438758542728232961)]} 0 1
[junit4:junit4] 2> 570815 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10026 (1438758542728232960)]} 0 1
[junit4:junit4] 2> 570819 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[26 (1438758542731378688)]} 0 0
[junit4:junit4] 2> 570820 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[26 (1438758542731378688)]} 0 3
[junit4:junit4] 2> 570820 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10026 (1438758542731378689)]} 0 1
[junit4:junit4] 2> 570821 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10026 (1438758542731378689)]} 0 4
[junit4:junit4] 2> 570881 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[27 (1438758542797438976)]} 0 1
[junit4:junit4] 2> 570882 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10027 (1438758542798487552)]} 0 1
[junit4:junit4] 2> 570886 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10027 (1438758542801633281)]} 0 0
[junit4:junit4] 2> 570886 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[27 (1438758542801633280)]} 0 0
[junit4:junit4] 2> 570887 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10027 (1438758542801633281)]} 0 3
[junit4:junit4] 2> 570887 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[27 (1438758542801633280)]} 0 4
[junit4:junit4] 2> 570915 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10028 (1438758542833090560)]} 0 1
[junit4:junit4] 2> 570915 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[28 (1438758542833090561)]} 0 1
[junit4:junit4] 2> 570920 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10028 (1438758542836236288)]} 0 1
[junit4:junit4] 2> 570920 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[28 (1438758542836236289)]} 0 1
[junit4:junit4] 2> 570920 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10028 (1438758542836236288)]} 0 3
[junit4:junit4] 2> 570920 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[28 (1438758542836236289)]} 0 3
[junit4:junit4] 2> 571016 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[29 (1438758542938996736)]} 0 1
[junit4:junit4] 2> 571016 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10029 (1438758542938996737)]} 0 1
[junit4:junit4] 2> 571021 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[29 (1438758542942142464)]} 0 1
[junit4:junit4] 2> 571021 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10029 (1438758542942142465)]} 0 0
[junit4:junit4] 2> 571021 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[29 (1438758542942142464)]} 0 3
[junit4:junit4] 2> 571022 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10029 (1438758542942142465)]} 0 4
[junit4:junit4] 2> 571026 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[30 (1438758542950531072)]} 0 1
[junit4:junit4] 2> 571026 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10030 (1438758542950531073)]} 0 0
[junit4:junit4] 2> 571031 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[30 (1438758542952628224)]} 0 1
[junit4:junit4] 2> 571031 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10030 (1438758542952628225)]} 0 0
[junit4:junit4] 2> 571031 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[30 (1438758542952628224)]} 0 3
[junit4:junit4] 2> 571032 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10030 (1438758542952628225)]} 0 4
[junit4:junit4] 2> 571046 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10029 (-1438758542971502592)]} 0 0
[junit4:junit4] 2> 571046 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[29 (-1438758542971502593)]} 0 0
[junit4:junit4] 2> 571050 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10029 (-1438758542973599744)]} 0 0
[junit4:junit4] 2> 571050 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[29 (-1438758542974648320)]} 0 0
[junit4:junit4] 2> 571051 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10029 (-1438758542973599744)]} 0 3
[junit4:junit4] 2> 571051 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[29 (-1438758542974648320)]} 0 3
[junit4:junit4] 2> 571053 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10031 (1438758542978842624)]} 0 0
[junit4:junit4] 2> 571054 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[31 (1438758542978842625)]} 0 1
[junit4:junit4] 2> 571058 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10031 (1438758542981988352)]} 0 0
[junit4:junit4] 2> 571059 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[31 (1438758542981988353)]} 0 1
[junit4:junit4] 2> 571059 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10031 (1438758542981988352)]} 0 4
[junit4:junit4] 2> 571059 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[31 (1438758542981988353)]} 0 4
[junit4:junit4] 2> 571093 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10030 (-1438758543019737088)]} 0 1
[junit4:junit4] 2> 571093 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[30 (-1438758543019737089)]} 0 1
[junit4:junit4] 2> 571097 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10030 (-1438758543022882816)]} 0 1
[junit4:junit4] 2> 571097 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[30 (-1438758543022882817)]} 0 1
[junit4:junit4] 2> 571097 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10030 (-1438758543022882816)]} 0 2
[junit4:junit4] 2> 571097 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[30 (-1438758543022882817)]} 0 2
[junit4:junit4] 2> 571100 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10032 (1438758543027077120)]} 0 1
[junit4:junit4] 2> 571100 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[32 (1438758543027077121)]} 0 1
[junit4:junit4] 2> 571104 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10032 (1438758543030222848)]} 0 0
[junit4:junit4] 2> 571105 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10032 (1438758543030222848)]} 0 4
[junit4:junit4] 2> 571105 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[32 (1438758543030222849)]} 0 1
[junit4:junit4] 2> 571106 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[32 (1438758543030222849)]} 0 4
[junit4:junit4] 2> 571126 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10033 (1438758543054340096)]} 0 1
[junit4:junit4] 2> 571126 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[33 (1438758543055388672)]} 0 0
[junit4:junit4] 2> 571130 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10033 (1438758543057485824)]} 0 0
[junit4:junit4] 2> 571131 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[33 (1438758543057485825)]} 0 1
[junit4:junit4] 2> 571131 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10033 (1438758543057485824)]} 0 3
[junit4:junit4] 2> 571131 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[33 (1438758543057485825)]} 0 3
[junit4:junit4] 2> 571177 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[31 (-1438758543107817472)]} 0 1
[junit4:junit4] 2> 571177 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10031 (-1438758543107817473)]} 0 1
[junit4:junit4] 2> 571181 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[31 (-1438758543110963200)]} 0 0
[junit4:junit4] 2> 571181 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10031 (-1438758543110963201)]} 0 0
[junit4:junit4] 2> 571181 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[31 (-1438758543110963200)]} 0 2
[junit4:junit4] 2> 571182 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10031 (-1438758543110963201)]} 0 3
[junit4:junit4] 2> 571184 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10034 (1438758543116206081)]} 0 0
[junit4:junit4] 2> 571185 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[34 (1438758543116206080)]} 0 1
[junit4:junit4] 2> 571189 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10034 (1438758543119351808)]} 0 0
[junit4:junit4] 2> 571190 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[34 (1438758543119351809)]} 0 1
[junit4:junit4] 2> 571190 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10034 (1438758543119351808)]} 0 3
[junit4:junit4] 2> 571190 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[34 (1438758543119351809)]} 0 3
[junit4:junit4] 2> 571274 T1754 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47680
[junit4:junit4] 2> 571274 T1754 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=13725951
[junit4:junit4] 2> 571276 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[35 (1438758543211626496)]} 0 1
[junit4:junit4] 2> 571276 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10035 (1438758543211626497)]} 0 1
[junit4:junit4] 2> 571281 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[35 (1438758543214772224)]} 0 0
[junit4:junit4] 2> 571282 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10035 (1438758543214772225)]} 0 1
[junit4:junit4] 2> 571282 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[35 (1438758543214772224)]} 0 4
[junit4:junit4] 2> 571282 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10035 (1438758543214772225)]} 0 4
[junit4:junit4] 2> 571354 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10036 (1438758543294464000)]} 0 0
[junit4:junit4] 2> 571355 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[36 (1438758543294464001)]} 0 1
[junit4:junit4] 2> 571359 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10036 (1438758543296561152)]} 0 0
[junit4:junit4] 2> 571360 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[36 (1438758543297609728)]} 0 1
[junit4:junit4] 2> 571360 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10036 (1438758543296561152)]} 0 4
[junit4:junit4] 2> 571360 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[36 (1438758543297609728)]} 0 4
[junit4:junit4] 2> 571370 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[33 (-1438758543311241216)]} 0 0
[junit4:junit4] 2> 571370 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10033 (-1438758543311241217)]} 0 0
[junit4:junit4] 2> 571375 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[33 (-1438758543314386944)]} 0 0
[junit4:junit4] 2> 571375 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10033 (-1438758543314386945)]} 0 0
[junit4:junit4] 2> 571375 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[33 (-1438758543314386944)]} 0 2
[junit4:junit4] 2> 571376 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10033 (-1438758543314386945)]} 0 3
[junit4:junit4] 2> 571378 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[37 (1438758543319629824)]} 0 1
[junit4:junit4] 2> 571378 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10037 (1438758543319629825)]} 0 1
[junit4:junit4] 2> 571383 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[37 (1438758543321726976)]} 0 1
[junit4:junit4] 2> 571383 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10037 (1438758543321726977)]} 0 1
[junit4:junit4] 2> 571384 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[37 (1438758543321726976)]} 0 4
[junit4:junit4] 2> 571384 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10037 (1438758543321726977)]} 0 4
[junit4:junit4] 2> 571430 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[34 (-1438758543374155776)]} 0 0
[junit4:junit4] 2> 571430 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10034 (-1438758543374155777)]} 0 0
[junit4:junit4] 2> 571434 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[34 (-1438758543377301504)]} 0 0
[junit4:junit4] 2> 571434 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10034 (-1438758543377301505)]} 0 0
[junit4:junit4] 2> 571435 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[34 (-1438758543377301504)]} 0 3
[junit4:junit4] 2> 571435 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10034 (-1438758543377301505)]} 0 3
[junit4:junit4] 2> 571438 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[38 (1438758543381495808)]} 0 1
[junit4:junit4] 2> 571438 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10038 (1438758543381495809)]} 0 1
[junit4:junit4] 2> 571442 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[38 (1438758543384641536)]} 0 0
[junit4:junit4] 2> 571443 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10038 (1438758543384641537)]} 0 1
[junit4:junit4] 2> 571443 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[38 (1438758543384641536)]} 0 3
[junit4:junit4] 2> 571443 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10038 (1438758543384641537)]} 0 3
[junit4:junit4] 2> 571464 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[39 (1438758543409807361)]} 0 0
[junit4:junit4] 2> 571464 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10039 (1438758543409807360)]} 0 0
[junit4:junit4] 2> 571469 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[39 (1438758543411904512)]} 0 1
[junit4:junit4] 2> 571469 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[39 (1438758543411904512)]} 0 3
[junit4:junit4] 2> 571469 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10039 (1438758543412953088)]} 0 0
[junit4:junit4] 2> 571470 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10039 (1438758543412953088)]} 0 4
[junit4:junit4] 2> 571560 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[37 (-1438758543510470656)]} 0 0
[junit4:junit4] 2> 571560 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10037 (-1438758543510470657)]} 0 0
[junit4:junit4] 2> 571564 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[37 (-1438758543512567808)]} 0 0
[junit4:junit4] 2> 571564 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10037 (-1438758543512567809)]} 0 0
[junit4:junit4] 2> 571564 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[37 (-1438758543512567808)]} 0 2
[junit4:junit4] 2> 571565 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10037 (-1438758543512567809)]} 0 3
[junit4:junit4] 2> 571567 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[40 (1438758543517810688)]} 0 0
[junit4:junit4] 2> 571567 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10040 (1438758543517810689)]} 0 0
[junit4:junit4] 2> 571572 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[40 (1438758543519907840)]} 0 1
[junit4:junit4] 2> 571572 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10040 (1438758543519907841)]} 0 0
[junit4:junit4] 2> 571572 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[40 (1438758543519907840)]} 0 3
[junit4:junit4] 2> 571573 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10040 (1438758543519907841)]} 0 4
[junit4:junit4] 2> 571585 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10039 (-1438758543536685056)]} 0 0
[junit4:junit4] 2> 571585 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[39 (-1438758543536685057)]} 0 0
[junit4:junit4] 2> 571589 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[39 (-1438758543539830784)]} 0 0
[junit4:junit4] 2> 571589 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10039 (-1438758543538782208)]} 0 0
[junit4:junit4] 2> 571590 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[39 (-1438758543539830784)]} 0 3
[junit4:junit4] 2> 571590 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10039 (-1438758543538782208)]} 0 3
[junit4:junit4] 2> 571592 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[41 (1438758543544025088)]} 0 0
[junit4:junit4] 2> 571593 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10041 (1438758543544025089)]} 0 1
[junit4:junit4] 2> 571597 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[41 (1438758543546122240)]} 0 1
[junit4:junit4] 2> 571597 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10041 (1438758543547170816)]} 0 0
[junit4:junit4] 2> 571597 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[41 (1438758543546122240)]} 0 3
[junit4:junit4] 2> 571598 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10041 (1438758543547170816)]} 0 3
[junit4:junit4] 2> 571656 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10042 (1438758543610085376)]} 0 1
[junit4:junit4] 2> 571656 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[42 (1438758543610085377)]} 0 1
[junit4:junit4] 2> 571661 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10042 (1438758543613231104)]} 0 1
[junit4:junit4] 2> 571661 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[42 (1438758543613231105)]} 0 1
[junit4:junit4] 2> 571662 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10042 (1438758543613231104)]} 0 4
[junit4:junit4] 2> 571662 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[42 (1438758543613231105)]} 0 4
[junit4:junit4] 2> 571756 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10042 (-1438758543715991553)]} 0 0
[junit4:junit4] 2> 571756 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[42 (-1438758543715991552)]} 0 0
[junit4:junit4] 2> 571760 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[42 (-1438758543719137280)]} 0 0
[junit4:junit4] 2> 571760 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10042 (-1438758543718088704)]} 0 0
[junit4:junit4] 2> 571761 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[42 (-1438758543719137280)]} 0 2
[junit4:junit4] 2> 571761 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10042 (-1438758543718088704)]} 0 3
[junit4:junit4] 2> 571764 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[43 (1438758543723331584)]} 0 1
[junit4:junit4] 2> 571764 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10043 (1438758543723331585)]} 0 1
[junit4:junit4] 2> 571769 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[43 (1438758543727525888)]} 0 0
[junit4:junit4] 2> 571770 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10043 (1438758543727525889)]} 0 1
[junit4:junit4] 2> 571770 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[43 (1438758543727525888)]} 0 4
[junit4:junit4] 2> 571770 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10043 (1438758543727525889)]} 0 4
[junit4:junit4] 2> 571864 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10043 (-1438758543829237761)]} 0 0
[junit4:junit4] 2> 571864 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[43 (-1438758543829237760)]} 0 0
[junit4:junit4] 2> 571868 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10043 (-1438758543832383488)]} 0 0
[junit4:junit4] 2> 571869 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[43 (-1438758543832383489)]} 0 1
[junit4:junit4] 2> 571869 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10043 (-1438758543832383488)]} 0 2
[junit4:junit4] 2> 571869 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[43 (-1438758543832383489)]} 0 2
[junit4:junit4] 2> 571871 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10044 (1438758543836577792)]} 0 0
[junit4:junit4] 2> 571872 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[44 (1438758543836577793)]} 0 1
[junit4:junit4] 2> 571876 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10044 (1438758543839723520)]} 0 1
[junit4:junit4] 2> 571876 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[44 (1438758543839723521)]} 0 0
[junit4:junit4] 2> 571876 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10044 (1438758543839723520)]} 0 3
[junit4:junit4] 2> 571877 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[44 (1438758543839723521)]} 0 3
[junit4:junit4] 2> 571880 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10045 (1438758543844966400)]} 0 1
[junit4:junit4] 2> 571880 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[45 (1438758543846014976)]} 0 1
[junit4:junit4] 2> 571884 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10045 (1438758543848112128)]} 0 0
[junit4:junit4] 2> 571884 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[45 (1438758543848112129)]} 0 0
[junit4:junit4] 2> 571885 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10045 (1438758543848112128)]} 0 3
[junit4:junit4] 2> 571885 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[45 (1438758543848112129)]} 0 3
[junit4:junit4] 2> 571966 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10044 (-1438758543936192512)]} 0 0
[junit4:junit4] 2> 571966 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[44 (-1438758543936192513)]} 0 0
[junit4:junit4] 2> 571970 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10044 (-1438758543938289664)]} 0 0
[junit4:junit4] 2> 571970 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[44 (-1438758543939338240)]} 0 0
[junit4:junit4] 2> 571971 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10044 (-1438758543938289664)]} 0 3
[junit4:junit4] 2> 571971 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[44 (-1438758543939338240)]} 0 3
[junit4:junit4] 2> 571973 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10046 (1438758543943532544)]} 0 1
[junit4:junit4] 2> 571973 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[46 (1438758543943532545)]} 0 0
[junit4:junit4] 2> 571978 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10046 (1438758543945629696)]} 0 1
[junit4:junit4] 2> 571978 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10046 (1438758543945629696)]} 0 3
[junit4:junit4] 2> 571978 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[46 (1438758543946678272)]} 0 0
[junit4:junit4] 2> 571979 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[46 (1438758543946678272)]} 0 4
[junit4:junit4] 2> 572022 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10047 (1438758543994912768)]} 0 1
[junit4:junit4] 2> 572022 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[47 (1438758543994912769)]} 0 0
[junit4:junit4] 2> 572028 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10047 (1438758543997009920)]} 0 1
[junit4:junit4] 2> 572029 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[47 (1438758543998058496)]} 0 2
[junit4:junit4] 2> 572029 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10047 (1438758543997009920)]} 0 5
[junit4:junit4] 2> 572029 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[47 (1438758543998058496)]} 0 4
[junit4:junit4] 2> 572119 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10048 (1438758544095576065)]} 0 1
[junit4:junit4] 2> 572119 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[48 (1438758544095576064)]} 0 1
[junit4:junit4] 2> 572125 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10048 (1438758544099770368)]} 0 1
[junit4:junit4] 2> 572125 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[48 (1438758544099770369)]} 0 0
[junit4:junit4] 2> 572126 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10048 (1438758544099770368)]} 0 5
[junit4:junit4] 2> 572126 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[48 (1438758544099770369)]} 0 4
[junit4:junit4] 2> 572216 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[49 (1438758544197287936)]} 0 1
[junit4:junit4] 2> 572216 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10049 (1438758544197287937)]} 0 1
[junit4:junit4] 2> 572221 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[49 (1438758544200433664)]} 0 1
[junit4:junit4] 2> 572221 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10049 (1438758544201482240)]} 0 0
[junit4:junit4] 2> 572221 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[49 (1438758544200433664)]} 0 3
[junit4:junit4] 2> 572222 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10049 (1438758544201482240)]} 0 3
[junit4:junit4] 2> 572274 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10050 (1438758544259153920)]} 0 0
[junit4:junit4] 2> 572274 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[50 (1438758544259153921)]} 0 0
[junit4:junit4] 2> 572280 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[50 (1438758544262299649)]} 0 1
[junit4:junit4] 2> 572280 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10050 (1438758544262299648)]} 0 1
[junit4:junit4] 2> 572280 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[50 (1438758544262299649)]} 0 3
[junit4:junit4] 2> 572280 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10050 (1438758544262299648)]} 0 3
[junit4:junit4] 2> 572284 T1815 C280 P47680 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:42558/z/r/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 572285 T1815 C280 P47680 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47680/z/r START replicas=[http://127.0.0.1:42558/z/r/collection1/] nUpdates=100
[junit4:junit4] 2> 572287 T1815 C280 P47680 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 572290 T1815 C280 P47680 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 572290 T1815 C280 P47680 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 572290 T1815 C280 P47680 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 572290 T1815 C280 P47680 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 572290 T1815 C280 P47680 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:42558/z/r/collection1/. core=collection1
[junit4:junit4] 2> 572291 T1815 C280 P47680 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 572292 T1791 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/get params={qt=/get&wt=javabin&distrib=false&getVersions=100&version=2} status=0 QTime=1
[junit4:junit4] 2> 572293 T1790 C278 P42558 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 572296 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[51 (1438758544281174016)]} 0 1
[junit4:junit4] 2> 572296 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10051 (1438758544281174017)]} 0 1
[junit4:junit4] 2> 572300 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[51 (1438758544284319744)]} 0 0
[junit4:junit4] 2> 572300 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10051 (1438758544284319745)]} 0 0
[junit4:junit4] 2> 572301 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[51 (1438758544284319744)]} 0 3
[junit4:junit4] 2> 572301 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10051 (1438758544284319745)]} 0 3
[junit4:junit4] 2> 572304 T1790 C278 P42558 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60049b lockFactory=org.apache.lucene.store.NativeFSLockFactory@114890b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60049b lockFactory=org.apache.lucene.store.NativeFSLockFactory@114890b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 572304 T1790 C278 P42558 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 572311 T1790 C278 P42558 oass.SolrIndexSearcher.<init> Opening Searcher@7cc304 realtime
[junit4:junit4] 2> 572311 T1790 C278 P42558 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 572311 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={commit=true&openSearcher=false&waitSearcher=true&wt=javabin&softCommit=false&version=2&commit_end_point=true} {commit=} 0 18
[junit4:junit4] 2> 572313 T1815 C280 P47680 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 572313 T1815 C280 P47680 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 572323 T1788 C278 P42558 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 572323 T1788 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&command=indexversion&wt=javabin&version=2} status=0 QTime=9
[junit4:junit4] 2> 572324 T1815 C280 P47680 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 572324 T1815 C280 P47680 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 572324 T1815 C280 P47680 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 572326 T1792 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&command=filelist&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 572327 T1815 C280 P47680 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 572327 T1815 C280 P47680 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index.20130624235111223
[junit4:junit4] 2> 572327 T1815 C280 P47680 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@d7bd03 lockFactory=org.apache.lucene.store.NativeFSLockFactory@14c9a94) fullCopy=false
[junit4:junit4] 2> 572329 T1790 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&file=_0.cfe&command=filecontent&wt=filestream&checksum=true&generation=2} status=0 QTime=0
[junit4:junit4] 2> 572332 T1791 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&file=_0.si&command=filecontent&wt=filestream&checksum=true&generation=2} status=0 QTime=0
[junit4:junit4] 2> 572334 T1788 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&file=_0_1.del&command=filecontent&wt=filestream&checksum=true&generation=2} status=0 QTime=0
[junit4:junit4] 2> 572337 T1792 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&file=_0.cfs&command=filecontent&wt=filestream&checksum=true&generation=2} status=0 QTime=0
[junit4:junit4] 2> 572340 T1790 C278 P42558 oasc.SolrCore.execute [collection1] webapp=/z/r path=/replication params={qt=/replication&file=segments_2&command=filecontent&wt=filestream&checksum=true&generation=2} status=0 QTime=0
[junit4:junit4] 2> 572341 T1815 C280 P47680 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 572342 T1815 C280 P47680 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 572343 T1815 C280 P47680 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 572343 T1815 C280 P47680 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 572353 T1815 C280 P47680 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_2,generation=2}
[junit4:junit4] 2> 572354 T1815 C280 P47680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 572356 T1815 C280 P47680 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_2,generation=2}
[junit4:junit4] 2> 572356 T1815 C280 P47680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 572356 T1815 C280 P47680 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 572357 T1815 C280 P47680 oass.SolrIndexSearcher.<init> Opening Searcher@122f071 main
[junit4:junit4] 2> 572358 T1814 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@122f071 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c90/36)}
[junit4:junit4] 2> 572359 T1815 C280 P47680 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index.20130624235111223 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index.20130624235111223;done=true>>]
[junit4:junit4] 2> 572359 T1815 C280 P47680 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index.20130624235111223
[junit4:junit4] 2> 572359 T1815 C280 P47680 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index.20130624235111223
[junit4:junit4] 2> 572359 T1815 C280 P47680 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 572359 T1823 C280 P47680 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=12958
[junit4:junit4] 2> 572363 T1823 C280 P47680 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 572367 T1823 C280 P47680 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8517c5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180ed10),segFN=segments_3,generation=3}
[junit4:junit4] 2> 572368 T1823 C280 P47680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 572370 T1823 C280 P47680 oass.SolrIndexSearcher.<init> Opening Searcher@45671e main
[junit4:junit4] 2> 572370 T1823 C280 P47680 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 572371 T1814 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45671e main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c90/36 _1(5.0):c2)}
[junit4:junit4] 2> 572371 T1823 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] {add=[51 (1438758544284319744), 10051 (1438758544284319745)]} 0 12
[junit4:junit4] 2> 572371 T1823 C280 P47680 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0 deleteByQuery=0 errors=0 positionOfStart=12958}
[junit4:junit4] 2> 572371 T1815 C280 P47680 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 572371 T1815 C280 P47680 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 572371 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[52 (1438758544360865793)]} 0 0
[junit4:junit4] 2> 572371 T1815 C280 P47680 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 572371 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10052 (1438758544360865792)]} 0 0
[junit4:junit4] 2> 572373 T1815 C280 P47680 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 572378 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[52 (1438758544364011520)]} 0 1
[junit4:junit4] 2> 572378 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10052 (1438758544365060096)]} 0 1
[junit4:junit4] 2> 572378 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[52 (1438758544364011520)]} 0 4
[junit4:junit4] 2> 572379 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10052 (1438758544365060096)]} 0 5
[junit4:junit4] 2> 572448 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[53 (1438758544440557568)]} 0 1
[junit4:junit4] 2> 572448 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10053 (1438758544440557569)]} 0 1
[junit4:junit4] 2> 572453 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[53 (1438758544443703296)]} 0 1
[junit4:junit4] 2> 572453 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10053 (1438758544443703297)]} 0 0
[junit4:junit4] 2> 572453 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[53 (1438758544443703296)]} 0 3
[junit4:junit4] 2> 572454 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10053 (1438758544443703297)]} 0 4
[junit4:junit4] 2> 572463 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[53 (-1438758544457334784)]} 0 0
[junit4:junit4] 2> 572463 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10053 (-1438758544457334785)]} 0 0
[junit4:junit4] 2> 572467 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[53 (-1438758544459431936)]} 0 0
[junit4:junit4] 2> 572467 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10053 (-1438758544460480512)]} 0 0
[junit4:junit4] 2> 572468 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[53 (-1438758544459431936)]} 0 3
[junit4:junit4] 2> 572468 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10053 (-1438758544460480512)]} 0 3
[junit4:junit4] 2> 572471 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[54 (1438758544464674816)]} 0 1
[junit4:junit4] 2> 572471 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10054 (1438758544464674817)]} 0 1
[junit4:junit4] 2> 572475 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[54 (1438758544467820544)]} 0 0
[junit4:junit4] 2> 572476 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[54 (1438758544467820544)]} 0 3
[junit4:junit4] 2> 572476 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10054 (1438758544467820545)]} 0 1
[junit4:junit4] 2> 572477 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10054 (1438758544467820545)]} 0 4
[junit4:junit4] 2> 572518 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[55 (1438758544513957888)]} 0 1
[junit4:junit4] 2> 572518 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10055 (1438758544515006464)]} 0 0
[junit4:junit4] 2> 572522 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[55 (1438758544517103616)]} 0 0
[junit4:junit4] 2> 572523 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[55 (1438758544517103616)]} 0 3
[junit4:junit4] 2> 572523 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10055 (1438758544517103617)]} 0 0
[junit4:junit4] 2> 572524 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10055 (1438758544517103617)]} 0 4
[junit4:junit4] 2> 572537 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[55 (-1438758544534929408)]} 0 0
[junit4:junit4] 2> 572538 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10055 (-1438758544535977984)]} 0 0
[junit4:junit4] 2> 572541 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[55 (-1438758544538075136)]} 0 0
[junit4:junit4] 2> 572542 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10055 (-1438758544538075137)]} 0 0
[junit4:junit4] 2> 572542 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[55 (-1438758544538075136)]} 0 3
[junit4:junit4] 2> 572542 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10055 (-1438758544538075137)]} 0 2
[junit4:junit4] 2> 572545 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10056 (1438758544542269441)]} 0 1
[junit4:junit4] 2> 572545 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[56 (1438758544542269440)]} 0 1
[junit4:junit4] 2> 572549 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10056 (1438758544545415168)]} 0 1
[junit4:junit4] 2> 572549 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10056 (1438758544545415168)]} 0 3
[junit4:junit4] 2> 572549 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[56 (1438758544545415169)]} 0 0
[junit4:junit4] 2> 572550 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[56 (1438758544545415169)]} 0 3
[junit4:junit4] 2> 572599 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10057 (1438758544598892544)]} 0 1
[junit4:junit4] 2> 572599 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[57 (1438758544598892545)]} 0 1
[junit4:junit4] 2> 572600 T1777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 572600 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47680/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:47680_z%2Fr",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 572602 T1777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47680/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:47680_z%2Fr",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 572604 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10057 (1438758544602038272)]} 0 1
[junit4:junit4] 2> 572604 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[57 (1438758544602038273)]} 0 1
[junit4:junit4] 2> 572604 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10057 (1438758544602038272)]} 0 3
[junit4:junit4] 2> 572605 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[57 (1438758544602038273)]} 0 4
[junit4:junit4] 2> 572606 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 572606 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 572606 T1783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 572606 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 572682 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[58 (1438758544685924353)]} 0 1
[junit4:junit4] 2> 572682 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10058 (1438758544685924352)]} 0 1
[junit4:junit4] 2> C280_STATE=coll:collection1 core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:47680/z/r, collection=collection1, state=active, node_name=127.0.0.1:47680_z%2Fr, core=collection1}
[junit4:junit4] 2> 572687 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[58 (1438758544689070080)]} 0 1
[junit4:junit4] 2> 572687 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10058 (1438758544689070081)]} 0 1
[junit4:junit4] 2> 572687 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[58 (1438758544689070080)]} 0 3
[junit4:junit4] 2> 572687 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10058 (1438758544689070081)]} 0 3
[junit4:junit4] 2> 572698 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[59 (1438758544702701568)]} 0 1
[junit4:junit4] 2> 572698 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10059 (1438758544703750144)]} 0 1
[junit4:junit4] 2> 572702 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[59 (1438758544705847296)]} 0 0
[junit4:junit4] 2> 572702 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10059 (1438758544705847297)]} 0 0
[junit4:junit4] 2> 572703 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[59 (1438758544705847296)]} 0 4
[junit4:junit4] 2> 572703 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10059 (1438758544705847297)]} 0 3
[junit4:junit4] 2> 572744 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[60 (1438758544751984640)]} 0 0
[junit4:junit4] 2> 572744 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10060 (1438758544751984641)]} 0 0
[junit4:junit4] 2> 572749 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[60 (1438758544755130368)]} 0 0
[junit4:junit4] 2> 572749 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[60 (1438758544755130368)]} 0 3
[junit4:junit4] 2> 572750 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10060 (1438758544755130369)]} 0 1
[junit4:junit4] 2> 572750 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10060 (1438758544755130369)]} 0 3
[junit4:junit4] 2> 572811 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[61 (1438758544821190656)]} 0 1
[junit4:junit4] 2> 572811 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10061 (1438758544821190657)]} 0 1
[junit4:junit4] 2> 572816 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[61 (1438758544825384960)]} 0 0
[junit4:junit4] 2> 572816 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10061 (1438758544825384961)]} 0 0
[junit4:junit4] 2> 572817 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[61 (1438758544825384960)]} 0 4
[junit4:junit4] 2> 572817 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10061 (1438758544825384961)]} 0 4
[junit4:junit4] 2> 572837 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[58 (-1438758544849502208)]} 0 0
[junit4:junit4] 2> 572837 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10058 (-1438758544849502209)]} 0 0
[junit4:junit4] 2> 572841 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[58 (-1438758544851599360)]} 0 0
[junit4:junit4] 2> 572842 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {delete=[10058 (-1438758544852647936)]} 0 1
[junit4:junit4] 2> 572842 T1791 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[58 (-1438758544851599360)]} 0 3
[junit4:junit4] 2> 572843 T1792 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {delete=[10058 (-1438758544852647936)]} 0 4
[junit4:junit4] 2> 572846 T1772 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[62 (1438758544857890816)]} 0 1
[junit4:junit4] 2> 572846 T1771 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10062 (1438758544857890817)]} 0 1
[junit4:junit4] 2> 572850 T1807 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[62 (1438758544861036544)]} 0 0
[junit4:junit4] 2> 572851 T1808 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10062 (1438758544861036545)]} 0 1
[junit4:junit4] 2> 572851 T1788 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[62 (1438758544861036544)]} 0 3
[junit4:junit4] 2> 572851 T1790 C278 P42558 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10062 (1438758544861036545)]} 0 3
[junit4:junit4] 2> 572913 T1768 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[10063 (1438758544928145408)]} 0 1
[junit4:junit4] 2> 572913 T1770 C279 P59100 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={wt=javabin&version=2} {add=[63 (1438758544928145409)]} 0 1
[junit4:junit4] 2> 572921 T1806 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:42558/z/r/collection1/} {add=[10063 (1438758544932339712)]} 0 2
[junit4:junit4] 2> 572921 T1805 C280 P47680 oasup.LogUpdateProcessor.finish [collection1] webapp=/z/r path=/u
[...truncated too long message...]
z/r/collection1/ has no replicas
[junit4:junit4] 2> 696926 T1837 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47680/z/r/collection1/
[junit4:junit4] 2> 696926 T1837 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 696928 T1849 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42558/z/r",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:42558_z%2Fr",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 696935 T1837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 696936 T1754 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/z/r,null}
[junit4:junit4] 2> 696987 T1754 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47680
[junit4:junit4] 2> 696987 T1754 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=15641079
[junit4:junit4] 2> 697124 T1776 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 697125 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 697125 T1776 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 697989 T1754 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 697989 T1754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 697990 T1754 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@ed2cfe
[junit4:junit4] 2> 697995 T1754 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=58,cumulative_deletesById=30,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 697997 T1754 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 697997 T1754 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 697998 T1754 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 697999 T1754 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 697999 T1754 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 698000 T1754 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index;done=false>>]
[junit4:junit4] 2> 698000 T1754 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2/index
[junit4:junit4] 2> 698000 T1754 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2;done=false>>]
[junit4:junit4] 2> 698000 T1754 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1372107056389/jetty2
[junit4:junit4] 2> 698001 T1849 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89922408048558090-127.0.0.1:47680_z%2Fr-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 698023 T1754 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/z/r,null}
[junit4:junit4] 2> 698077 T1754 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 698079 T1754 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50294 50294
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=A1C5ACADBDB94B8E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_ME_#Latn -Dtests.timezone=Turkey -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 141s J1 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<704> but was:<58>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([A1C5ACADBDB94B8E:202322B5CAE62BB2]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 698116 T1754 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 140664 T1753 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 698215 T1796 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 698215 T1796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 698215 T1796 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 699302 T1837 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 699302 T1837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 699302 T1837 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=sr_ME_#Latn, timezone=Turkey
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=138651200,total=283312128
[junit4:junit4] 2> NOTE: All tests run in this JVM: [OpenCloseCoreStressTest, SpellCheckCollatorTest, TestDFRSimilarityFactory, TestIndexSearcher, TestCollationField, DirectSolrSpellCheckerTest, TestOmitPositions, TestSolrXMLSerializer, TestDistributedGrouping, HighlighterConfigTest, URLClassifyProcessorTest, DateMathParserTest, BadCopyFieldTest, TestReload, TestLMJelinekMercerSimilarityFactory, TestRecovery, TestPostingsSolrHighlighter, TestMergePolicyConfig, HighlighterTest, TimeZoneUtilsTest, TestAnalyzedSuggestions, AliasIntegrationTest, ConvertedLegacyTest, TestClassNameShortening, ClusterStateTest, TestFieldTypeCollectionResource, DocValuesMultiTest, TestAtomicUpdateErrorCases, DocumentAnalysisRequestHandlerTest, ShardRoutingCustomTest, UpdateParamsTest, TestSystemIdResolver, RequiredFieldsTest, TestIndexingPerformance, DOMUtilTest, BasicDistributedZk2Test, SuggesterWFSTTest, SignatureUpdateProcessorFactoryTest, TestSolrXmlPersistence, EchoParamsTest, BadIndexSchemaTest, TestSchemaVersionResource, LeaderElectionIntegrationTest, TestHashPartitioner, TestMultiCoreConfBootstrap, SortByFunctionTest, SolrRequestParserTest, TestWordDelimiterFilterFactory, MultiTermTest, TestRangeQuery, TestFiltering, TestReloadAndDeleteDocs, CurrencyFieldOpenExchangeTest, ZkNodePropsTest, NoCacheHeaderTest, TestJmxIntegration, SolrCmdDistributorTest, TestDefaultSearchFieldResource, OutputWriterTest, TestBinaryField, PingRequestHandlerTest, DistanceFunctionTest, TestQuerySenderNoQuery, OpenExchangeRatesOrgProviderTest, FileUtilsTest, DirectUpdateHandlerTest, XsltUpdateRequestHandlerTest, TestCSVLoader, TestCoreDiscovery, TestDistributedSearch, TestZkChroot, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, BasicZkTest, RecoveryZkTest]
[junit4:junit4] Completed on J1 in 142.15s, 1 test, 1 failure <<< FAILURES!
[...truncated 447 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 36 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
6295 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6295/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
2 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:42617/_aet/i/collection1lastClient and got 244 from http://127.0.0.1:40026/_aet/i/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:42617/_aet/i/collection1lastClient and got 244 from http://127.0.0.1:40026/_aet/i/collection1
at __randomizedtesting.SeedInfo.seed([4D7FA934FCBB6311:CC99272C8BE4032D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
FAILED: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:33775/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:33775/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([4D7FA934FCBB6311:CC99272C8BE4032D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
Build Log:
[...truncated 9311 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 365630 T558 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 365634 T558 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737
[junit4:junit4] 2> 365635 T558 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 365635 T559 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 365736 T558 oasc.ZkTestServer.run start zk server on port:51935
[junit4:junit4] 2> 365737 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 365826 T565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90354fcc name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 365826 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 365827 T558 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 365831 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 365832 T567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fcebcfb name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 365833 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 365833 T558 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 365836 T558 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 365838 T558 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 365840 T558 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 365842 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 365842 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 365845 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 365846 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 365849 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 365849 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 365851 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 365852 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 365854 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 365855 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 365857 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 365858 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 365860 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 365860 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 365863 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 365863 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 365865 T558 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 365866 T558 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 365965 T558 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 365970 T558 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44481
[junit4:junit4] 2> 365971 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 365972 T558 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 365973 T558 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970
[junit4:junit4] 2> 365974 T558 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/solr.xml
[junit4:junit4] 2> 365975 T558 oasc.CoreContainer.<init> New CoreContainer -1693870417
[junit4:junit4] 2> 365976 T558 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/'
[junit4:junit4] 2> 365976 T558 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/'
[junit4:junit4] 2> 366018 T558 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 366019 T558 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 366019 T558 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 366020 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 366021 T558 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 366021 T558 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 366022 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 366023 T558 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 366023 T558 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 366024 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 366029 T558 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 366030 T558 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51935/solr
[junit4:junit4] 2> 366031 T558 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 366032 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366034 T578 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6496e634 name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366034 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366036 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 366041 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366042 T580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d8f5999b name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366043 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366044 T558 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 366046 T558 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 366049 T558 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 366051 T558 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44481_
[junit4:junit4] 2> 366052 T558 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44481_
[junit4:junit4] 2> 366054 T558 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 366057 T558 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 366059 T558 oasc.Overseer.start Overseer (id=89923057204527107-127.0.0.1:44481_-n_0000000000) starting
[junit4:junit4] 2> 366062 T558 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 366065 T582 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 366065 T558 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 366067 T558 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 366069 T558 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 366072 T581 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 366075 T583 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 366075 T583 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 367575 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 367576 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44481",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44481_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 367576 T581 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 367576 T581 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 367579 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 368077 T583 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/collection1
[junit4:junit4] 2> 368077 T583 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 368078 T583 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 368079 T583 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 368080 T583 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/collection1/'
[junit4:junit4] 2> 368081 T583 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/collection1/lib/README' to classloader
[junit4:junit4] 2> 368082 T583 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 368118 T583 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 368149 T583 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 368150 T583 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 368155 T583 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 368477 T583 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 368482 T583 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 368485 T583 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 368496 T583 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 368500 T583 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 368504 T583 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 368505 T583 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 368506 T583 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 368506 T583 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 368507 T583 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 368507 T583 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 368508 T583 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 368508 T583 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372116961970/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/
[junit4:junit4] 2> 368508 T583 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c1c8e6e3
[junit4:junit4] 2> 368509 T583 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 368509 T583 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data
[junit4:junit4] 2> 368510 T583 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/index/
[junit4:junit4] 2> 368510 T583 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 368511 T583 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/index
[junit4:junit4] 2> 368513 T583 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b30d02c lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d8ffbdb),segFN=segments_1,generation=1}
[junit4:junit4] 2> 368513 T583 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 368515 T583 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 368515 T583 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 368516 T583 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 368516 T583 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 368517 T583 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 368517 T583 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 368518 T583 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 368518 T583 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 368519 T583 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 368520 T583 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 368522 T583 oass.SolrIndexSearcher.<init> Opening Searcher@d903269f main
[junit4:junit4] 2> 368523 T583 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/tlog
[junit4:junit4] 2> 368523 T583 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 368524 T583 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 368527 T584 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d903269f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 368528 T583 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 368528 T583 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44481 collection:control_collection shard:shard1
[junit4:junit4] 2> 368529 T583 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 368535 T583 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 368537 T583 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 368537 T583 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 368538 T583 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44481/collection1/
[junit4:junit4] 2> 368538 T583 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 368538 T583 oasc.SyncStrategy.syncToMe http://127.0.0.1:44481/collection1/ has no replicas
[junit4:junit4] 2> 368539 T583 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44481/collection1/
[junit4:junit4] 2> 368539 T583 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 369083 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 369089 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 369095 T583 oasc.ZkController.register We are http://127.0.0.1:44481/collection1/ and leader is http://127.0.0.1:44481/collection1/
[junit4:junit4] 2> 369095 T583 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44481
[junit4:junit4] 2> 369095 T583 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 369096 T583 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 369096 T583 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 369097 T583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 369098 T558 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 369099 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 369101 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 369108 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 369110 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369112 T587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bab9a06 name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369112 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369113 T558 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 369116 T558 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 369210 T558 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 369212 T558 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48651
[junit4:junit4] 2> 369213 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 369214 T558 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 369215 T558 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218
[junit4:junit4] 2> 369215 T558 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/solr.xml
[junit4:junit4] 2> 369216 T558 oasc.CoreContainer.<init> New CoreContainer 1870490669
[junit4:junit4] 2> 369217 T558 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/'
[junit4:junit4] 2> 369218 T558 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/'
[junit4:junit4] 2> 369269 T558 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 369270 T558 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 369271 T558 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 369272 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 369272 T558 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 369273 T558 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 369274 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 369275 T558 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 369275 T558 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 369276 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 369282 T558 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 369283 T558 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51935/solr
[junit4:junit4] 2> 369284 T558 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 369285 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369287 T598 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dd3ff43 name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369287 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369289 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 369294 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369295 T600 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26916285 name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369296 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369299 T558 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 370302 T558 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48651_
[junit4:junit4] 2> 370303 T558 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48651_
[junit4:junit4] 2> 370305 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 370306 T587 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 370306 T600 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 370307 T580 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 370310 T601 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 370311 T601 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 370593 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 370594 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44481",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44481__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:44481_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 370598 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 370598 T581 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 370599 T581 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 370603 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 370603 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 370603 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 371312 T601 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/collection1
[junit4:junit4] 2> 371313 T601 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 371314 T601 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 371314 T601 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 371316 T601 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/collection1/'
[junit4:junit4] 2> 371317 T601 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/collection1/lib/README' to classloader
[junit4:junit4] 2> 371318 T601 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 371356 T601 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 371401 T601 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 371402 T601 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 371407 T601 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 371740 T601 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 371747 T601 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 371749 T601 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 371761 T601 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 371765 T601 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 371769 T601 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 371770 T601 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 371771 T601 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 371771 T601 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 371772 T601 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 371773 T601 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 371773 T601 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 371773 T601 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/
[junit4:junit4] 2> 371774 T601 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c1c8e6e3
[junit4:junit4] 2> 371774 T601 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 371775 T601 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1
[junit4:junit4] 2> 371775 T601 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/index/
[junit4:junit4] 2> 371776 T601 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 371776 T601 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/index
[junit4:junit4] 2> 371778 T601 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3579f449 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c6e716e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 371778 T601 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 371780 T601 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 371781 T601 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 371781 T601 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 371782 T601 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 371783 T601 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 371783 T601 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 371783 T601 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 371784 T601 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 371784 T601 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 371786 T601 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 371788 T601 oass.SolrIndexSearcher.<init> Opening Searcher@a1e3bcad main
[junit4:junit4] 2> 371789 T601 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/tlog
[junit4:junit4] 2> 371789 T601 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 371790 T601 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 371795 T602 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a1e3bcad main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 371797 T601 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 371797 T601 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48651 collection:collection1 shard:shard1
[junit4:junit4] 2> 371798 T601 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 371802 T601 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 371804 T601 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 371805 T601 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 371805 T601 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48651/collection1/
[junit4:junit4] 2> 371805 T601 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 371806 T601 oasc.SyncStrategy.syncToMe http://127.0.0.1:48651/collection1/ has no replicas
[junit4:junit4] 2> 371806 T601 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48651/collection1/
[junit4:junit4] 2> 371806 T601 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 372107 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 372114 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 372114 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 372114 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 372162 T601 oasc.ZkController.register We are http://127.0.0.1:48651/collection1/ and leader is http://127.0.0.1:48651/collection1/
[junit4:junit4] 2> 372162 T601 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48651
[junit4:junit4] 2> 372162 T601 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 372163 T601 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 372163 T601 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 372165 T601 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 372166 T558 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 372167 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 372168 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 372246 T558 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 372248 T558 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33775
[junit4:junit4] 2> 372249 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 372250 T558 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 372250 T558 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274
[junit4:junit4] 2> 372251 T558 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/solr.xml
[junit4:junit4] 2> 372251 T558 oasc.CoreContainer.<init> New CoreContainer -1880142678
[junit4:junit4] 2> 372252 T558 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/'
[junit4:junit4] 2> 372253 T558 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/'
[junit4:junit4] 2> 372297 T558 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 372298 T558 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 372298 T558 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 372299 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 372300 T558 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 372300 T558 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 372301 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 372301 T558 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 372302 T558 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 372303 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 372308 T558 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 372309 T558 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51935/solr
[junit4:junit4] 2> 372309 T558 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 372310 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 372312 T614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c89bbfd name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 372312 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 372313 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 372318 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 372320 T616 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fbe1f06 name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 372320 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 372323 T558 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 373326 T558 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33775_
[junit4:junit4] 2> 373327 T558 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33775_
[junit4:junit4] 2> 373330 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 373330 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 373330 T587 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 373331 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 373330 T616 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 373331 T580 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 373332 T600 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 373336 T617 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 373336 T617 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 373620 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 373621 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48651__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 373624 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33775",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33775_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 373624 T581 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 373624 T581 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 373628 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 373628 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 373628 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 373628 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 374337 T617 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/collection1
[junit4:junit4] 2> 374338 T617 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 374338 T617 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 374339 T617 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 374340 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/collection1/'
[junit4:junit4] 2> 374341 T617 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/collection1/lib/README' to classloader
[junit4:junit4] 2> 374341 T617 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 374366 T617 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 374396 T617 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 374398 T617 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 374402 T617 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 374714 T617 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 374721 T617 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 374724 T617 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 374737 T617 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 374740 T617 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 374744 T617 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 374745 T617 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 374745 T617 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 374746 T617 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 374747 T617 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 374747 T617 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 374748 T617 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 374748 T617 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372116968274/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/
[junit4:junit4] 2> 374748 T617 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c1c8e6e3
[junit4:junit4] 2> 374749 T617 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 374750 T617 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2
[junit4:junit4] 2> 374750 T617 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/index/
[junit4:junit4] 2> 374750 T617 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 374751 T617 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/index
[junit4:junit4] 2> 374753 T617 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6690a524 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a4e216b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 374753 T617 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 374755 T617 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 374755 T617 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 374756 T617 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 374756 T617 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 374757 T617 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 374757 T617 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 374758 T617 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 374758 T617 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 374759 T617 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 374760 T617 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 374762 T617 oass.SolrIndexSearcher.<init> Opening Searcher@d0c46688 main
[junit4:junit4] 2> 374762 T617 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/tlog
[junit4:junit4] 2> 374763 T617 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 374763 T617 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 374769 T618 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d0c46688 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 374771 T617 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 374772 T617 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33775 collection:collection1 shard:shard2
[junit4:junit4] 2> 374773 T617 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 374778 T617 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 374780 T617 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 374781 T617 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 374781 T617 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33775/collection1/
[junit4:junit4] 2> 374781 T617 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 374782 T617 oasc.SyncStrategy.syncToMe http://127.0.0.1:33775/collection1/ has no replicas
[junit4:junit4] 2> 374782 T617 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33775/collection1/
[junit4:junit4] 2> 374783 T617 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 375133 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375138 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 375138 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 375138 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 375138 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 375188 T617 oasc.ZkController.register We are http://127.0.0.1:33775/collection1/ and leader is http://127.0.0.1:33775/collection1/
[junit4:junit4] 2> 375188 T617 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33775
[junit4:junit4] 2> 375189 T617 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 375189 T617 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 375189 T617 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 375191 T617 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375192 T558 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 375192 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 375193 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 375273 T558 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 375275 T558 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33246
[junit4:junit4] 2> 375275 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 375276 T558 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 375277 T558 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299
[junit4:junit4] 2> 375277 T558 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/solr.xml
[junit4:junit4] 2> 375278 T558 oasc.CoreContainer.<init> New CoreContainer 1054187138
[junit4:junit4] 2> 375279 T558 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/'
[junit4:junit4] 2> 375279 T558 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/'
[junit4:junit4] 2> 375318 T558 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 375319 T558 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 375319 T558 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 375320 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 375320 T558 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 375321 T558 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 375322 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 375322 T558 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 375323 T558 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 375323 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 375328 T558 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 375328 T558 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51935/solr
[junit4:junit4] 2> 375329 T558 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 375330 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 375331 T630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@363e3b13 name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 375332 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 375333 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 375338 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 375339 T632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d7c2e88 name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 375339 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 375342 T558 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 376344 T558 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33246_
[junit4:junit4] 2> 376346 T558 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33246_
[junit4:junit4] 2> 376348 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 376348 T616 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 376348 T587 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 376348 T632 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 376348 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 376349 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 376349 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 376350 T580 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 376351 T600 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 376354 T633 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 376354 T633 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 376643 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 376644 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33775",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33775__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33775_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 376648 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33246",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33246_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 376648 T581 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 376648 T581 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 376651 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 376651 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 376651 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 376651 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 376651 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 377356 T633 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/collection1
[junit4:junit4] 2> 377357 T633 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 377358 T633 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 377359 T633 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 377361 T633 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/collection1/'
[junit4:junit4] 2> 377362 T633 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/collection1/lib/README' to classloader
[junit4:junit4] 2> 377363 T633 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 377397 T633 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 377429 T633 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 377430 T633 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 377435 T633 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 377759 T633 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 377765 T633 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 377767 T633 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 377778 T633 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 377782 T633 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 377785 T633 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 377786 T633 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 377786 T633 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 377787 T633 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 377788 T633 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 377788 T633 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 377788 T633 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 377789 T633 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372116971299/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/
[junit4:junit4] 2> 377789 T633 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c1c8e6e3
[junit4:junit4] 2> 377790 T633 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 377790 T633 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3
[junit4:junit4] 2> 377791 T633 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index/
[junit4:junit4] 2> 377791 T633 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 377791 T633 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index
[junit4:junit4] 2> 377793 T633 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cc70fe3d lockFactory=org.apache.lucene.store.NativeFSLockFactory@428f8938),segFN=segments_1,generation=1}
[junit4:junit4] 2> 377793 T633 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 377795 T633 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 377795 T633 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 377796 T633 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 377797 T633 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 377797 T633 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 377798 T633 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 377798 T633 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 377799 T633 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 377799 T633 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 377800 T633 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 377802 T633 oass.SolrIndexSearcher.<init> Opening Searcher@156a7dc1 main
[junit4:junit4] 2> 377803 T633 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/tlog
[junit4:junit4] 2> 377804 T633 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 377804 T633 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 377809 T634 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@156a7dc1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 377810 T633 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 377811 T633 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33246 collection:collection1 shard:shard1
[junit4:junit4] 2> 377814 T633 oasc.ZkController.register We are http://127.0.0.1:33246/collection1/ and leader is http://127.0.0.1:48651/collection1/
[junit4:junit4] 2> 377814 T633 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33246
[junit4:junit4] 2> 377814 T633 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 377814 T633 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C172 name=collection1 org.apache.solr.core.SolrCore@b5e8d2c3 url=http://127.0.0.1:33246/collection1 node=127.0.0.1:33246_ C172_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:33246, state=down, node_name=127.0.0.1:33246_, collection=collection1, core=collection1}
[junit4:junit4] 2> 377815 T635 C172 P33246 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 377815 T633 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 377815 T635 C172 P33246 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 377816 T635 C172 P33246 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 377816 T635 C172 P33246 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 377819 T558 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 377820 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 377819 T635 C172 P33246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 377821 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 377826 T595 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:33246__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 377923 T558 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 377926 T558 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48437
[junit4:junit4] 2> 377927 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 377928 T558 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 377929 T558 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930
[junit4:junit4] 2> 377930 T558 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/solr.xml
[junit4:junit4] 2> 377931 T558 oasc.CoreContainer.<init> New CoreContainer 1733236424
[junit4:junit4] 2> 377932 T558 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/'
[junit4:junit4] 2> 377933 T558 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/'
[junit4:junit4] 2> 377987 T558 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 377988 T558 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 377989 T558 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 377989 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 377990 T558 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 377990 T558 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 377991 T558 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 377991 T558 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 377992 T558 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 377993 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 377998 T558 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 377998 T558 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51935/solr
[junit4:junit4] 2> 377999 T558 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 378000 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 378002 T647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e02a5b94 name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 378002 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 378003 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 378008 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 378010 T649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dc8e164 name:ZooKeeperConnection Watcher:127.0.0.1:51935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 378011 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 378014 T558 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 378155 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 378156 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33246",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33246__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33246_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 378158 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 378158 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 378158 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 378158 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 378159 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 378158 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 378827 T595 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:33246__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 378828 T595 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:33246__collection1&wt=javabin&nodeName=127.0.0.1:33246_&core=collection1} status=0 QTime=1002
[junit4:junit4] 2> 379016 T558 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48437_
[junit4:junit4] 2> 379018 T558 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48437_
[junit4:junit4] 2> 379020 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 379020 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 379021 T649 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 379020 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 379020 T616 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 379021 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379022 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379022 T632 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 379022 T580 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 379021 T587 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 379023 T600 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 379023 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379030 T650 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 379030 T650 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 379663 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 379664 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48437",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48437_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 379664 T581 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 379664 T581 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 379668 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379668 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379668 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379668 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379668 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379668 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 380032 T650 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/collection1
[junit4:junit4] 2> 380032 T650 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 380033 T650 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 380033 T650 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 380035 T650 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/collection1/'
[junit4:junit4] 2> 380036 T650 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/collection1/lib/README' to classloader
[junit4:junit4] 2> 380037 T650 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 380062 T650 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 380099 T650 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 380100 T650 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 380105 T650 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 380411 T650 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 380423 T650 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 380427 T650 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 380437 T650 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 380441 T650 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 380445 T650 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 380446 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 380446 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 380447 T650 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 380448 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 380448 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 380448 T650 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 380449 T650 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372116973930/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/
[junit4:junit4] 2> 380449 T650 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c1c8e6e3
[junit4:junit4] 2> 380450 T650 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 380450 T650 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4
[junit4:junit4] 2> 380451 T650 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index/
[junit4:junit4] 2> 380451 T650 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 380452 T650 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index
[junit4:junit4] 2> 380453 T650 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492fd39b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4dc6acc6),segFN=segments_1,generation=1}
[junit4:junit4] 2> 380454 T650 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 380456 T650 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 380456 T650 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 380457 T650 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 380457 T650 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 380458 T650 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 380458 T650 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 380459 T650 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 380459 T650 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 380460 T650 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 380461 T650 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 380463 T650 oass.SolrIndexSearcher.<init> Opening Searcher@d206f567 main
[junit4:junit4] 2> 380464 T650 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/tlog
[junit4:junit4] 2> 380465 T650 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 380465 T650 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 380469 T651 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d206f567 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 380471 T650 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 380471 T650 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48437 collection:collection1 shard:shard2
[junit4:junit4] 2> 380474 T650 oasc.ZkController.register We are http://127.0.0.1:48437/collection1/ and leader is http://127.0.0.1:33775/collection1/
[junit4:junit4] 2> 380474 T650 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48437
[junit4:junit4] 2> 380474 T650 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 380475 T650 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C173 name=collection1 org.apache.solr.core.SolrCore@5e65535e url=http://127.0.0.1:48437/collection1 node=127.0.0.1:48437_ C173_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:48437, state=down, node_name=127.0.0.1:48437_, collection=collection1, core=collection1}
[junit4:junit4] 2> 380475 T652 C173 P48437 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 380475 T650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 380475 T652 C173 P48437 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 380476 T652 C173 P48437 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 380476 T652 C173 P48437 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 380476 T558 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 380477 T558 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 380477 T652 C173 P48437 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 380478 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 380486 T558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 380487 T611 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:48437__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 380487 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 380496 T593 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> ASYNC NEW_CORE C174 name=collection1 org.apache.solr.core.SolrCore@b5e8d2c3 url=http://127.0.0.1:33246/collection1 node=127.0.0.1:33246_ C174_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:33246, state=recovering, node_name=127.0.0.1:33246_, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 380830 T635 C174 P33246 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:48651/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 380830 T635 C174 P33246 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33246 START replicas=[http://127.0.0.1:48651/collection1/] nUpdates=100
[junit4:junit4] 2> 380830 T635 C174 P33246 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 380831 T635 C174 P33246 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 380831 T635 C174 P33246 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 380831 T635 C174 P33246 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 380831 T635 C174 P33246 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 380832 T635 C174 P33246 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:48651/collection1/. core=collection1
[junit4:junit4] 2> 380832 T635 C174 P33246 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C175 name=collection1 org.apache.solr.core.SolrCore@9e5ceebc url=http://127.0.0.1:48651/collection1 node=127.0.0.1:48651_ C175_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:48651, state=active, node_name=127.0.0.1:48651_, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 380835 T595 C175 P48651 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 380837 T595 C175 P48651 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 380839 T595 C175 P48651 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3579f449 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c6e716e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 380839 T595 C175 P48651 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 380840 T595 C175 P48651 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3579f449 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c6e716e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3579f449 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3c6e716e),segFN=segments_2,generation=2}
[junit4:junit4] 2> 380841 T595 C175 P48651 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 380841 T595 C175 P48651 oass.SolrIndexSearcher.<init> Opening Searcher@288428a3 realtime
[junit4:junit4] 2> 380842 T595 C175 P48651 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 380843 T595 C175 P48651 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 6
[junit4:junit4] 2> 380844 T635 C174 P33246 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 380844 T635 C174 P33246 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 380846 T594 C175 P48651 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 380846 T594 C175 P48651 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 380847 T635 C174 P33246 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 380847 T635 C174 P33246 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 380847 T635 C174 P33246 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 380849 T595 C175 P48651 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 380850 T635 C174 P33246 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 380851 T635 C174 P33246 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index.20130625083616952
[junit4:junit4] 2> 380851 T635 C174 P33246 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@b407965e lockFactory=org.apache.lucene.store.NativeFSLockFactory@ef4dae0e) fullCopy=false
[junit4:junit4] 2> 380853 T596 C175 P48651 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 380854 T635 C174 P33246 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 380855 T635 C174 P33246 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 380855 T635 C174 P33246 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 380857 T635 C174 P33246 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cc70fe3d lockFactory=org.apache.lucene.store.NativeFSLockFactory@428f8938),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cc70fe3d lockFactory=org.apache.lucene.store.NativeFSLockFactory@428f8938),segFN=segments_2,generation=2}
[junit4:junit4] 2> 380857 T635 C174 P33246 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 380857 T635 C174 P33246 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 380858 T635 C174 P33246 oass.SolrIndexSearcher.<init> Opening Searcher@1d2ed8ff main
[junit4:junit4] 2> 380859 T634 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d2ed8ff main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 380860 T635 C174 P33246 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index.20130625083616952 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index.20130625083616952;done=true>>]
[junit4:junit4] 2> 380860 T635 C174 P33246 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index.20130625083616952
[junit4:junit4] 2> 380860 T635 C174 P33246 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index.20130625083616952
[junit4:junit4] 2> 380860 T635 C174 P33246 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 380861 T635 C174 P33246 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 380861 T635 C174 P33246 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 380861 T635 C174 P33246 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 380862 T635 C174 P33246 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 381173 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 381173 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48437",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48437__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48437_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 381176 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 381176 T581 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 381177 T581 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 381180 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33246",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33246__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33246_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 381182 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 381182 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 381182 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 381182 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 381182 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 381182 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 381488 T611 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:48437__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 381488 T611 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:48437__collection1&wt=javabin&nodeName=127.0.0.1:48437_&core=collection1} status=0 QTime=1001
[junit4:junit4] 2> 381498 T593 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/onenodecollectioncore
[junit4:junit4] 2> 381499 T593 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 381500 T593 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 381500 T593 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 381502 T593 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 381502 T593 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 381505 T593 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 381506 T593 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/onenodecollectioncore/'
[junit4:junit4] 2> 381531 T593 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 381563 T593 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 381565 T593 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 381569 T593 oass.IndexSchema.readSchema [onenodecollectioncore] Schema name=test
[junit4:junit4] 2> 381892 T593 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 381899 T593 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 381901 T593 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 381914 T593 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 381918 T593 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 381922 T593 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 381923 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 381923 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 381924 T593 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 381925 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 381925 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 381926 T593 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 381927 T593 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372116965218/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/
[junit4:junit4] 2> 381927 T593 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c1c8e6e3
[junit4:junit4] 2> 381928 T593 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 381929 T593 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection
[junit4:junit4] 2> 381930 T593 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/index/
[junit4:junit4] 2> 381930 T593 oasc.SolrCore.initIndex WARN [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 381931 T593 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/index
[junit4:junit4] 2> 381933 T593 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a112b537 lockFactory=org.apache.lucene.store.NativeFSLockFactory@99ef2feb),segFN=segments_1,generation=1}
[junit4:junit4] 2> 381933 T593 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 381935 T593 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 381936 T593 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 381937 T593 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 381938 T593 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 381938 T593 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 381939 T593 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 381940 T593 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 381940 T593 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 381941 T593 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 381943 T593 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 381945 T593 oass.SolrIndexSearcher.<init> Opening Searcher@2b32f5e9 main
[junit4:junit4] 2> 381946 T593 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/tlog
[junit4:junit4] 2> 381947 T593 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 381947 T593 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 381950 T656 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@2b32f5e9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 381952 T593 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 381953 T593 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:48651 collection:onenodecollection shard:shard1
[junit4:junit4] 2> 381954 T593 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 381958 T593 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 381960 T593 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 381961 T593 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 381961 T593 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48651/onenodecollectioncore/
[junit4:junit4] 2> 381962 T593 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 381963 T593 oasc.SyncStrategy.syncToMe http://127.0.0.1:48651/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 381963 T593 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48651/onenodecollectioncore/
[junit4:junit4] 2> 381964 T593 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 382688 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 382697 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 382697 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 382697 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 382697 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 382697 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 382697 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 382720 T593 oasc.ZkController.register We are http://127.0.0.1:48651/onenodecollectioncore/ and leader is http://127.0.0.1:48651/onenodecollectioncore/
[junit4:junit4] 2> 382720 T593 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:48651
[junit4:junit4] 2> 382721 T593 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 382722 T593 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 382723 T593 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 382724 T593 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection&action=CREATE&version=2&numShards=1&name=onenodecollectioncore&wt=javabin&roles=none&collection=onenodecollection} status=0 QTime=2228
[junit4:junit4] 2> 382725 T558 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 382726 T558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C176 name=collection1 org.apache.solr.core.SolrCore@5e65535e url=http://127.0.0.1:48437/collection1 node=127.0.0.1:48437_ C176_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:48437, state=recovering, node_name=127.0.0.1:48437_, collection=collection1, shard=shard2, core=collection1}
[junit4:junit4] 2> 383489 T652 C176 P48437 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33775/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 383490 T652 C176 P48437 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48437 START replicas=[http://127.0.0.1:33775/collection1/] nUpdates=100
[junit4:junit4] 2> 383490 T652 C176 P48437 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 383490 T652 C176 P48437 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 383491 T652 C176 P48437 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 383491 T652 C176 P48437 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 383491 T652 C176 P48437 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 383492 T652 C176 P48437 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33775/collection1/. core=collection1
[junit4:junit4] 2> 383492 T652 C176 P48437 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C177 name=collection1 org.apache.solr.core.SolrCore@d8cb1b6 url=http://127.0.0.1:33775/collection1 node=127.0.0.1:33775_ C177_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:33775, state=active, node_name=127.0.0.1:33775_, collection=collection1, shard=shard2, core=collection1, leader=true}
[junit4:junit4] 2> 383496 T611 C177 P33775 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 383499 T612 C177 P33775 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 383500 T612 C177 P33775 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6690a524 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a4e216b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 383501 T612 C177 P33775 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 383502 T612 C177 P33775 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6690a524 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a4e216b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6690a524 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2a4e216b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 383502 T612 C177 P33775 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 383503 T612 C177 P33775 oass.SolrIndexSearcher.<init> Opening Searcher@f88e91b5 realtime
[junit4:junit4] 2> 383504 T612 C177 P33775 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 383504 T612 C177 P33775 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 6
[junit4:junit4] 2> 383505 T652 C176 P48437 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 383505 T652 C176 P48437 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 383507 T609 C177 P33775 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 383507 T609 C177 P33775 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 383508 T652 C176 P48437 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 383508 T652 C176 P48437 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 383509 T652 C176 P48437 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 383510 T611 C177 P33775 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 383511 T652 C176 P48437 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 383512 T652 C176 P48437 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index.20130625083619613
[junit4:junit4] 2> 383512 T652 C176 P48437 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@f91bb18c lockFactory=org.apache.lucene.store.NativeFSLockFactory@38b62b35) fullCopy=false
[junit4:junit4] 2> 383514 T612 C177 P33775 oasc.SolrCore.execute [collection1] webapp= path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 383515 T652 C176 P48437 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 383516 T652 C176 P48437 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 383516 T652 C176 P48437 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 383517 T652 C176 P48437 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492fd39b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4dc6acc6),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@492fd39b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4dc6acc6),segFN=segments_2,generation=2}
[junit4:junit4] 2> 383518 T652 C176 P48437 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 383518 T652 C176 P48437 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 383518 T652 C176 P48437 oass.SolrIndexSearcher.<init> Opening Searcher@af775a83 main
[junit4:junit4] 2> 383520 T651 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@af775a83 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 383520 T652 C176 P48437 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index.20130625083619613 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index.20130625083619613;done=true>>]
[junit4:junit4] 2> 383521 T652 C176 P48437 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index.20130625083619613
[junit4:junit4] 2> 383521 T652 C176 P48437 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index.20130625083619613
[junit4:junit4] 2> 383521 T652 C176 P48437 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 383521 T652 C176 P48437 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 383522 T652 C176 P48437 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 383522 T652 C176 P48437 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 383523 T652 C176 P48437 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 383727 T558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 384204 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 384204 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48651__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 384207 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48437",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48437__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48437_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 384210 T587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 384210 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 384210 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 384210 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 384210 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 384210 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 384729 T558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 384731 T558 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 384732 T558 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 384755 T558 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 384756 T660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3af70a1c name:ZooKeeperConnection Watcher:127.0.0.1:51935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 384757 T558 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 384792 T558 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 44481
[junit4:junit4] 2> 384793 T558 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1693870417
[junit4:junit4] 2> 385715 T581 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 385716 T581 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44481",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44481__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:44481_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 385720 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 385720 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 385720 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 385721 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 385720 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 386794 T558 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 386795 T558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 386797 T558 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@8fcf08dc
[junit4:junit4] 2> 386804 T558 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 386805 T558 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 386806 T558 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 386807 T558 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 386808 T558 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 386809 T558 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 386810 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/index;done=false>>]
[junit4:junit4] 2> 386811 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data/index
[junit4:junit4] 2> 386811 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data;done=false>>]
[junit4:junit4] 2> 386812 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/control/data
[junit4:junit4] 2> 386813 T581 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89923057204527107-127.0.0.1:44481_-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 386813 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 386814 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 386814 T580 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 386815 T580 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 386814 T649 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 386815 T580 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 386815 T600 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 386815 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 386814 T616 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 386815 T632 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 386817 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 386817 T600 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 386819 T600 oasc.Overseer.start Overseer (id=89923057204527110-127.0.0.1:48651_-n_0000000001) starting
[junit4:junit4] 2> 386822 T662 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 386822 T661 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 386823 T661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 386824 T661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44481",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44481__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:44481_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 386826 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 386826 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 386826 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 386826 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 386834 T558 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 386887 T558 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 48651
[junit4:junit4] 2> 386887 T558 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1870490669
[junit4:junit4] 2> 388330 T661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 388331 T661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48651__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 388334 T661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48651__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 388338 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 388338 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 388338 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 388338 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 389890 T558 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 389891 T558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 389895 T558 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 389897 T558 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@9e5ceebc
[junit4:junit4] 2> 389905 T558 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 389906 T558 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 389907 T558 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 389908 T558 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 389910 T558 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 389912 T558 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 389913 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1;done=false>>]
[junit4:junit4] 2> 389914 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1
[junit4:junit4] 2> 389915 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/index;done=false>>]
[junit4:junit4] 2> 389916 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty1/index
[junit4:junit4] 2> 389917 T558 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@82d4b0b0
[junit4:junit4] 2> 389923 T558 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 389924 T558 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 389924 T558 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 389925 T558 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 389925 T558 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 389926 T558 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 389926 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/index;done=false>>]
[junit4:junit4] 2> 389927 T558 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection/index
[junit4:junit4] 2> 389928 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection;done=false>>]
[junit4:junit4] 2> 389928 T558 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372116961737/onenodecollection
[junit4:junit4] 2> 389929 T661 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89923057204527110-127.0.0.1:48651_-n_0000000001) am no longer a leader.
[junit4:junit4] 2> 389930 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 389930 T600 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 389930 T600 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 389930 T649 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 389931 T616 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 389931 T632 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 389931 T600 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 389931 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 389931 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 389932 T632 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 389934 T616 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 389934 T632 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 389934 T632 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 389935 T632 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 389935 T632 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33246/collection1/
[junit4:junit4] 2> 389935 T632 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 389935 T616 oasc.Overseer.start Overseer (id=89923057204527112-127.0.0.1:33775_-n_0000000002) starting
[junit4:junit4] 2> 389936 T632 oasc.SyncStrategy.syncToMe http://127.0.0.1:33246/collection1/ has no replicas
[junit4:junit4] 2> 389936 T632 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33246/collection1/
[junit4:junit4] 2> 389936 T632 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 389939 T664 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 389939 T663 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 389941 T663 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 389941 T558 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 389941 T663 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48651__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 389943 T663 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48651__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 389950 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 389950 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 389950 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 389992 T558 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 33775
[junit4:junit4] 2> 389994 T558 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1880142678
[junit4:junit4] 2> 390997 T558 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 390997 T558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 390999 T558 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@d8cb1b6
[junit4:junit4] 2> 391003 T558 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 391004 T558 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 391004 T558 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 391005 T558 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 391006 T558 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 391007 T558 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 391007 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2;done=false>>]
[junit4:junit4] 2> 391008 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2
[junit4:junit4] 2> 391008 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/index;done=false>>]
[junit4:junit4] 2> 391009 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty2/index
[junit4:junit4] 2> 391010 T663 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89923057204527112-127.0.0.1:33775_-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 391011 T649 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 391011 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 391011 T632 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 391013 T649 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 391014 T632 oasc.Overseer.start Overseer (id=89923057204527114-127.0.0.1:33246_-n_0000000003) starting
[junit4:junit4] 2> 391015 T649 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 391015 T649 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 391016 T649 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 391016 T649 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48437/collection1/
[junit4:junit4] 2> 391016 T649 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 391017 T649 oasc.SyncStrategy.syncToMe http://127.0.0.1:48437/collection1/ has no replicas
[junit4:junit4] 2> 391017 T649 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48437/collection1/
[junit4:junit4] 2> 391017 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 391017 T649 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 391017 T665 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 391017 T666 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 391019 T632 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 391020 T665 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 391021 T665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33775",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33775__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33775_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 391023 T665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33775",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33775__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33775_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 391031 T558 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 391031 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 391031 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 391083 T558 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 33246
[junit4:junit4] 2> 391084 T558 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1054187138
[junit4:junit4] 2> 392085 T558 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 392086 T558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 392088 T558 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@b5e8d2c3
[junit4:junit4] 2> 392092 T558 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 392092 T558 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 392093 T558 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 392093 T558 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 392094 T558 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 392095 T558 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 392096 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3;done=false>>]
[junit4:junit4] 2> 392096 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3
[junit4:junit4] 2> 392097 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index;done=false>>]
[junit4:junit4] 2> 392097 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty3/index
[junit4:junit4] 2> 392098 T665 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89923057204527114-127.0.0.1:33246_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 392099 T632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 392100 T632 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 392100 T632 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 392100 T649 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 392102 T649 oasc.Overseer.start Overseer (id=89923057204527116-127.0.0.1:48437_-n_0000000004) starting
[junit4:junit4] 2> 392106 T649 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 392106 T668 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 392106 T667 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 392106 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 392108 T667 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 392109 T667 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33246",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33246__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33246_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 392111 T667 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33246",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33246__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:33246_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 392114 T649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 392120 T558 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 392172 T558 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 48437
[junit4:junit4] 2> 392172 T558 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1733236424
[junit4:junit4] 2> 392311 T616 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 392311 T616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 392311 T616 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 393174 T558 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 393175 T558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 393183 T558 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@5e65535e
[junit4:junit4] 2> 393189 T558 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 393190 T558 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 393190 T558 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 393191 T558 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 393192 T558 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 393193 T558 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 393193 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4;done=false>>]
[junit4:junit4] 2> 393194 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4
[junit4:junit4] 2> 393194 T558 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index;done=false>>]
[junit4:junit4] 2> 393195 T558 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372116961736/jetty4/index
[junit4:junit4] 2> 393195 T667 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89923057204527116-127.0.0.1:48437_-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 393218 T558 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 393277 T558 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 393278 T558 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51935 51935
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89923057204527107-127.0.0.1:44481_-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (5)
[junit4:junit4] 1> /solr/overseer_elect/election/89923057204527112-127.0.0.1:33775_-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89923057204527114-127.0.0.1:33246_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89923057204527107-127.0.0.1:44481_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89923057204527116-127.0.0.1:48437_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89923057204527110-127.0.0.1:48651_-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89923057204527107-127.0.0.1:44481__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44481",
[junit4:junit4] 1> "node_name":"127.0.0.1:44481_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89923057204527110-127.0.0.1:48651__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89923057204527114-127.0.0.1:33246__collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89923057204527112-127.0.0.1:33775__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89923057204527116-127.0.0.1:48437__collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 1> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:33775",
[junit4:junit4] 1> "node_name":"127.0.0.1:33775_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89923057204527110-127.0.0.1:48651__onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:48651",
[junit4:junit4] 1> "node_name":"127.0.0.1:48651_",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (5)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:48437_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:33246_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:48651_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:33775_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:44481_ (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:44481__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44481",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:44481_",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "onenodecollection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff
[...truncated too long message...]
ns/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41969/_aet/i",
[junit4:junit4] 1> "node_name":"127.0.0.1:41969__aet%2Fi",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89923073135280140-127.0.0.1:36984__aet%2Fi_collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89923073135280138-127.0.0.1:40026__aet%2Fi_collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89923073135280142-127.0.0.1:42617__aet%2Fi_collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40026/_aet/i",
[junit4:junit4] 1> "node_name":"127.0.0.1:40026__aet%2Fi",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:36984__aet%2Fi (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40026__aet%2Fi (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:41969__aet%2Fi (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:42617__aet%2Fi (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:41969__aet%2Fi_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41969/_aet/i",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:41969__aet%2Fi",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:42617__aet%2Fi_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:42617/_aet/i",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:42617__aet%2Fi",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"},
[junit4:junit4] 1> "127.0.0.1:60449__aet%2Fi_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60449/_aet/i",
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "node_name":"127.0.0.1:60449__aet%2Fi",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"},
[junit4:junit4] 1> "127.0.0.1:40026__aet%2Fi_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40026/_aet/i",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:40026__aet%2Fi",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:36984__aet%2Fi_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:36984/_aet/i",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:36984__aet%2Fi",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=4D7FA934FCBB6311 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_LU_PREEURO -Dtests.timezone=Asia/Jayapura -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 101s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:42617/_aet/i/collection1lastClient and got 244 from http://127.0.0.1:40026/_aet/i/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4D7FA934FCBB6311:CC99272C8BE4032D]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2> 709481 T2393 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 100756 T2392 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=fr_LU_PREEURO, timezone=Asia/Jayapura
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=48103720,total=179175424
[junit4:junit4] 2> NOTE: All tests run in this JVM: [OverseerTest, TestIBSimilarityFactory, TestLFUCache, TestSolrIndexConfig, TestFastLRUCache, TestClassNameShortening, TestPseudoReturnFields, OverseerCollectionProcessorTest, PluginInfoTest, WordBreakSolrSpellCheckerTest, ShardRoutingTest, TestQueryUtils, TestLRUCache, BadComponentTest, MoreLikeThisHandlerTest, PreAnalyzedFieldTest, MBeansHandlerTest, TestSchemaResource, TestSolrQueryParser, DirectUpdateHandlerOptimizeTest, BadCopyFieldTest, TestCoreContainer, LoggingHandlerTest, TestDocumentBuilder, TestRandomDVFaceting, TestJoin, TestFuzzyAnalyzedSuggestions, SuggesterTest, IndexSchemaTest, CircularListTest, TestExtendedDismaxParser, ScriptEngineTest, RAMDirectoryFactoryTest, TestFunctionQuery, TestConfig, ShowFileRequestHandlerTest, PreAnalyzedUpdateProcessorTest, ResponseLogComponentTest, TestFaceting, TestUniqueKeyFieldResource, DocValuesTest, UnloadDistributedZkTest, ClusterStateUpdateTest, TestSolrCoreProperties, AutoCommitTest, TestMaxScoreQueryParser, TestDocSet, SystemInfoHandlerTest, TestSolrDeletionPolicy1, SearchHandlerTest, SolrInfoMBeanTest, TestWriterPerf, TestCloudManagedSchemaAddField, SpatialFilterTest, AlternateDirectoryTest, QueryEqualityTest, JsonLoaderTest, SchemaVersionSpecificBehaviorTest, TestReplicationHandler, TestFieldCollectionResource, TestSort, SimpleFacetsTest, ReturnFieldsTest, SynonymTokenizerTest, TestValueSourceCache, TestStressRecovery, TestSweetSpotSimilarityFactory, TestCharFilters, TestSearchPerf, PolyFieldTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 101.28s, 1 test, 1 failure <<< FAILURES!
[...truncated 470 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:379: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 error, 1 failure, 14 ignored (8 assumptions)
Total time: 40 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
6294 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6294/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
1 tests failed.
FAILED: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:47018/_/nx/onenodecollectioncore returned non ok status:404, message:Can not find: /_/nx/onenodecollectioncore/update
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:47018/_/nx/onenodecollectioncore returned non ok status:404, message:Can not find: /_/nx/onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([475CA3DF2F55AE9F:C6BA2DC7580ACEA3]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
Build Log:
[...truncated 9267 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 201184 T625 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/nx
[junit4:junit4] 2> 201190 T625 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587
[junit4:junit4] 2> 201192 T625 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 201193 T626 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 201293 T625 oasc.ZkTestServer.run start zk server on port:44835
[junit4:junit4] 2> 201294 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 201595 T632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cbf30b86 name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 201595 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 201596 T625 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 201600 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 201601 T634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e9180ba3 name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 201601 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 201602 T625 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 201606 T625 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 201609 T625 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 201610 T625 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 201613 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 201613 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 201616 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 201617 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 201619 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 201620 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 201622 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 201623 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 201624 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 201625 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 201627 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 201628 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 201630 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 201630 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 201632 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 201633 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 201635 T625 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 201636 T625 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 201727 T625 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 201731 T625 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35055
[junit4:junit4] 2> 201732 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 201733 T625 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 201733 T625 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034
[junit4:junit4] 2> 201734 T625 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/solr.xml
[junit4:junit4] 2> 201735 T625 oasc.CoreContainer.<init> New CoreContainer 1132056162
[junit4:junit4] 2> 201735 T625 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/'
[junit4:junit4] 2> 201736 T625 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/'
[junit4:junit4] 2> 201773 T625 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 201773 T625 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 201774 T625 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 201775 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 201775 T625 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 201776 T625 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 201776 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 201777 T625 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 201777 T625 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 201778 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 201783 T625 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 201783 T625 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44835/solr
[junit4:junit4] 2> 201784 T625 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 201785 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 201786 T645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27112224 name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 201787 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 201788 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 201793 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 201794 T647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@97d32f1 name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 201794 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 201796 T625 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 201799 T625 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 201801 T625 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 201803 T625 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35055__%2Fnx
[junit4:junit4] 2> 201804 T625 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35055__%2Fnx
[junit4:junit4] 2> 201807 T625 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 201811 T625 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 201813 T625 oasc.Overseer.start Overseer (id=89922706708299779-127.0.0.1:35055__%2Fnx-n_0000000000) starting
[junit4:junit4] 2> 201817 T625 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 201820 T649 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 201820 T625 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 201822 T625 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 201824 T625 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 201827 T648 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 201829 T650 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 201829 T650 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 203330 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 203331 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35055/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35055__%2Fnx",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 203331 T648 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 203331 T648 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 203335 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 203830 T650 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/collection1
[junit4:junit4] 2> 203831 T650 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 203831 T650 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 203832 T650 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 203833 T650 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/collection1/'
[junit4:junit4] 2> 203834 T650 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/collection1/lib/README' to classloader
[junit4:junit4] 2> 203834 T650 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 203858 T650 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 203898 T650 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 203899 T650 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 203904 T650 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 204205 T650 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 204211 T650 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 204213 T650 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 204225 T650 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 204231 T650 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 204236 T650 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 204238 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 204238 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 204239 T650 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 204240 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 204240 T650 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 204241 T650 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 204241 T650 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372111614034/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/
[junit4:junit4] 2> 204242 T650 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@be4b0b85
[junit4:junit4] 2> 204242 T650 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 204243 T650 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data
[junit4:junit4] 2> 204244 T650 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/index/
[junit4:junit4] 2> 204244 T650 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 204245 T650 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/index
[junit4:junit4] 2> 204247 T650 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e999b246 lockFactory=org.apache.lucene.store.NativeFSLockFactory@605e2a99; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 204248 T650 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 204250 T650 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 204250 T650 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 204251 T650 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 204252 T650 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 204253 T650 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 204253 T650 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 204254 T650 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 204254 T650 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 204255 T650 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 204269 T650 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 204271 T650 oass.SolrIndexSearcher.<init> Opening Searcher@fdf01d0f main
[junit4:junit4] 2> 204272 T650 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/tlog
[junit4:junit4] 2> 204273 T650 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 204273 T650 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 204276 T651 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fdf01d0f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 204277 T650 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 204278 T650 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35055/_/nx collection:control_collection shard:shard1
[junit4:junit4] 2> 204279 T650 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 204286 T650 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 204289 T650 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 204289 T650 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 204290 T650 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35055/_/nx/collection1/
[junit4:junit4] 2> 204290 T650 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 204291 T650 oasc.SyncStrategy.syncToMe http://127.0.0.1:35055/_/nx/collection1/ has no replicas
[junit4:junit4] 2> 204291 T650 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35055/_/nx/collection1/
[junit4:junit4] 2> 204292 T650 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 204840 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 204854 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 204897 T650 oasc.ZkController.register We are http://127.0.0.1:35055/_/nx/collection1/ and leader is http://127.0.0.1:35055/_/nx/collection1/
[junit4:junit4] 2> 204898 T650 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35055/_/nx
[junit4:junit4] 2> 204898 T650 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 204899 T650 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 204899 T650 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 204901 T650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 204904 T625 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 204905 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 204906 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 204912 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 204914 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 204916 T654 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ce6befe6 name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 204916 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 204918 T625 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 204920 T625 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 205025 T625 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 205028 T625 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43587
[junit4:junit4] 2> 205030 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 205030 T625 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 205031 T625 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317
[junit4:junit4] 2> 205032 T625 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/solr.xml
[junit4:junit4] 2> 205033 T625 oasc.CoreContainer.<init> New CoreContainer 135591031
[junit4:junit4] 2> 205034 T625 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/'
[junit4:junit4] 2> 205035 T625 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/'
[junit4:junit4] 2> 205092 T625 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 205093 T625 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 205094 T625 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 205094 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 205095 T625 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 205096 T625 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 205096 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 205097 T625 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 205097 T625 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 205098 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 205103 T625 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 205104 T625 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44835/solr
[junit4:junit4] 2> 205105 T625 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 205106 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 205108 T665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d5b754e name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 205108 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 205109 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 205117 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 205119 T667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@224acc12 name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 205119 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 205123 T625 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 206126 T625 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43587__%2Fnx
[junit4:junit4] 2> 206128 T625 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43587__%2Fnx
[junit4:junit4] 2> 206130 T647 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 206130 T667 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 206130 T654 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 206130 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 206135 T668 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 206135 T668 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 206364 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 206365 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35055/_/nx",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35055__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:35055__%2Fnx",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 206367 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 206368 T648 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 206368 T648 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 206371 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 206371 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 206371 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 207136 T668 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/collection1
[junit4:junit4] 2> 207137 T668 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 207138 T668 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 207138 T668 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 207139 T668 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/collection1/'
[junit4:junit4] 2> 207140 T668 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/collection1/lib/README' to classloader
[junit4:junit4] 2> 207141 T668 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 207179 T668 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 207225 T668 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 207227 T668 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 207233 T668 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 207693 T668 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 207698 T668 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 207701 T668 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 207712 T668 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 207716 T668 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 207720 T668 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 207721 T668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 207722 T668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 207722 T668 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 207723 T668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 207723 T668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 207724 T668 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 207724 T668 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/
[junit4:junit4] 2> 207724 T668 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@be4b0b85
[junit4:junit4] 2> 207725 T668 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 207726 T668 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1
[junit4:junit4] 2> 207726 T668 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/index/
[junit4:junit4] 2> 207726 T668 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 207727 T668 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/index
[junit4:junit4] 2> 207728 T668 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@17d2c538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30eb7ba0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 207728 T668 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 207730 T668 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 207730 T668 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 207731 T668 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 207732 T668 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 207732 T668 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 207732 T668 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 207733 T668 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 207733 T668 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 207734 T668 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 207735 T668 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 207736 T668 oass.SolrIndexSearcher.<init> Opening Searcher@1e4db6c2 main
[junit4:junit4] 2> 207737 T668 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/tlog
[junit4:junit4] 2> 207738 T668 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 207738 T668 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 207742 T669 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e4db6c2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 207743 T668 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 207744 T668 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43587/_/nx collection:collection1 shard:shard1
[junit4:junit4] 2> 207744 T668 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 207750 T668 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 207752 T668 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 207752 T668 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 207753 T668 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43587/_/nx/collection1/
[junit4:junit4] 2> 207753 T668 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 207753 T668 oasc.SyncStrategy.syncToMe http://127.0.0.1:43587/_/nx/collection1/ has no replicas
[junit4:junit4] 2> 207754 T668 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43587/_/nx/collection1/
[junit4:junit4] 2> 207754 T668 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 207876 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 207882 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 207882 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 207882 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 207909 T668 oasc.ZkController.register We are http://127.0.0.1:43587/_/nx/collection1/ and leader is http://127.0.0.1:43587/_/nx/collection1/
[junit4:junit4] 2> 207909 T668 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43587/_/nx
[junit4:junit4] 2> 207910 T668 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 207910 T668 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 207910 T668 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 207912 T668 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 207913 T625 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 207914 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 207915 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 207999 T625 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 208002 T625 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47018
[junit4:junit4] 2> 208002 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 208003 T625 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 208004 T625 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317
[junit4:junit4] 2> 208004 T625 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/solr.xml
[junit4:junit4] 2> 208005 T625 oasc.CoreContainer.<init> New CoreContainer -1763847281
[junit4:junit4] 2> 208006 T625 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/'
[junit4:junit4] 2> 208006 T625 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/'
[junit4:junit4] 2> 208053 T625 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 208054 T625 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 208055 T625 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 208056 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 208056 T625 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 208057 T625 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 208058 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 208059 T625 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 208059 T625 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 208060 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 208066 T625 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 208067 T625 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44835/solr
[junit4:junit4] 2> 208067 T625 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 208069 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 208070 T681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cb7e2fc name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 208071 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 208072 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 208077 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 208079 T683 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e297aaf name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 208079 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 208082 T625 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 209085 T625 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47018__%2Fnx
[junit4:junit4] 2> 209087 T625 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47018__%2Fnx
[junit4:junit4] 2> 209089 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 209090 T683 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 209090 T654 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 209090 T647 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 209090 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 209091 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 209092 T667 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 209096 T684 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 209097 T684 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 209389 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 209390 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:43587__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 209392 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47018/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47018__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 209393 T648 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 209393 T648 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 209396 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 209397 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 209397 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 209397 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 210098 T684 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/collection1
[junit4:junit4] 2> 210099 T684 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 210100 T684 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 210100 T684 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 210101 T684 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/collection1/'
[junit4:junit4] 2> 210102 T684 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/collection1/lib/README' to classloader
[junit4:junit4] 2> 210103 T684 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 210129 T684 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 210158 T684 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 210159 T684 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 210163 T684 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 210545 T684 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 210550 T684 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 210553 T684 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 210563 T684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 210566 T684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 210570 T684 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 210571 T684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 210571 T684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 210572 T684 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 210573 T684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 210573 T684 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 210573 T684 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 210574 T684 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372111620317/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/
[junit4:junit4] 2> 210574 T684 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@be4b0b85
[junit4:junit4] 2> 210575 T684 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 210575 T684 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2
[junit4:junit4] 2> 210576 T684 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/index/
[junit4:junit4] 2> 210576 T684 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 210576 T684 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/index
[junit4:junit4] 2> 210578 T684 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@d0aec391 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48271ad4; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 210578 T684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 210580 T684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 210580 T684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 210581 T684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 210581 T684 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 210582 T684 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 210582 T684 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 210582 T684 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 210583 T684 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 210583 T684 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 210585 T684 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 210587 T684 oass.SolrIndexSearcher.<init> Opening Searcher@249de8b4 main
[junit4:junit4] 2> 210587 T684 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/tlog
[junit4:junit4] 2> 210588 T684 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 210588 T684 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 210592 T685 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@249de8b4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 210593 T684 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 210594 T684 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47018/_/nx collection:collection1 shard:shard2
[junit4:junit4] 2> 210594 T684 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 210601 T684 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 210604 T684 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 210604 T684 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 210604 T684 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47018/_/nx/collection1/
[junit4:junit4] 2> 210605 T684 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 210605 T684 oasc.SyncStrategy.syncToMe http://127.0.0.1:47018/_/nx/collection1/ has no replicas
[junit4:junit4] 2> 210605 T684 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47018/_/nx/collection1/
[junit4:junit4] 2> 210606 T684 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 210903 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 210908 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 210908 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 210908 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 210909 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 210961 T684 oasc.ZkController.register We are http://127.0.0.1:47018/_/nx/collection1/ and leader is http://127.0.0.1:47018/_/nx/collection1/
[junit4:junit4] 2> 210961 T684 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47018/_/nx
[junit4:junit4] 2> 210961 T684 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 210962 T684 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 210962 T684 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 210964 T684 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 210964 T625 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 210965 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 210966 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 211035 T625 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 211037 T625 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57727
[junit4:junit4] 2> 211038 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 211038 T625 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 211039 T625 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366
[junit4:junit4] 2> 211039 T625 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/solr.xml
[junit4:junit4] 2> 211040 T625 oasc.CoreContainer.<init> New CoreContainer -2023196345
[junit4:junit4] 2> 211041 T625 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/'
[junit4:junit4] 2> 211041 T625 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/'
[junit4:junit4] 2> 211077 T625 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 211078 T625 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 211079 T625 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 211079 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 211080 T625 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 211080 T625 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 211081 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 211082 T625 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 211082 T625 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 211083 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 211087 T625 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 211088 T625 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44835/solr
[junit4:junit4] 2> 211088 T625 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 211089 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 211092 T697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c80e3735 name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 211092 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 211094 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 211099 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 211100 T699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b607c575 name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 211101 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 211103 T625 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 212106 T625 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57727__%2Fnx
[junit4:junit4] 2> 212107 T625 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57727__%2Fnx
[junit4:junit4] 2> 212109 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 212109 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 212110 T647 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 212110 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 212110 T654 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 212111 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 212111 T683 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 212110 T699 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 212112 T667 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 212116 T700 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 212116 T700 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 212413 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 212414 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47018/_/nx",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:47018__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47018__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 212417 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57727/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57727__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 212417 T648 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 212417 T648 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 212420 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 212421 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 212421 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 212421 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 212421 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 213118 T700 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/collection1
[junit4:junit4] 2> 213118 T700 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 213119 T700 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 213119 T700 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 213121 T700 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/collection1/'
[junit4:junit4] 2> 213121 T700 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/collection1/lib/README' to classloader
[junit4:junit4] 2> 213122 T700 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 213153 T700 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 213182 T700 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 213184 T700 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 213188 T700 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 213519 T700 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 213524 T700 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 213527 T700 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 213536 T700 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 213540 T700 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 213544 T700 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 213545 T700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 213546 T700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 213546 T700 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 213547 T700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 213547 T700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 213548 T700 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 213548 T700 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372111623366/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/
[junit4:junit4] 2> 213549 T700 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@be4b0b85
[junit4:junit4] 2> 213550 T700 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 213551 T700 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3
[junit4:junit4] 2> 213551 T700 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index/
[junit4:junit4] 2> 213552 T700 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 213552 T700 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index
[junit4:junit4] 2> 213554 T700 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@128778e5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60bf049a; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 213555 T700 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 213557 T700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 213557 T700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 213558 T700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 213559 T700 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 213560 T700 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 213560 T700 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 213561 T700 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 213562 T700 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 213562 T700 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 213564 T700 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 213566 T700 oass.SolrIndexSearcher.<init> Opening Searcher@6617447b main
[junit4:junit4] 2> 213567 T700 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/tlog
[junit4:junit4] 2> 213568 T700 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 213568 T700 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 213573 T701 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6617447b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 213575 T700 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 213575 T700 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57727/_/nx collection:collection1 shard:shard1
[junit4:junit4] 2> 213579 T700 oasc.ZkController.register We are http://127.0.0.1:57727/_/nx/collection1/ and leader is http://127.0.0.1:43587/_/nx/collection1/
[junit4:junit4] 2> 213579 T700 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57727/_/nx
[junit4:junit4] 2> 213579 T700 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 213580 T700 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C213 name=collection1 org.apache.solr.core.SolrCore@1bcd57e2 url=http://127.0.0.1:57727/_/nx/collection1 node=127.0.0.1:57727__%2Fnx C213_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:57727/_/nx, state=down, node_name=127.0.0.1:57727__%2Fnx, collection=collection1, core=collection1}
[junit4:junit4] 2> 213581 T702 C213 P57727 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 213581 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 213581 T702 C213 P57727 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 213582 T702 C213 P57727 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 213582 T702 C213 P57727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 213582 T625 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 213583 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 213583 T702 C213 P57727 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 213584 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 213590 T660 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:57727__%2Fnx_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 213661 T625 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 213663 T625 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41723
[junit4:junit4] 2> 213664 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 213665 T625 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 213665 T625 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987
[junit4:junit4] 2> 213666 T625 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/solr.xml
[junit4:junit4] 2> 213666 T625 oasc.CoreContainer.<init> New CoreContainer 716364007
[junit4:junit4] 2> 213667 T625 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/'
[junit4:junit4] 2> 213668 T625 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/'
[junit4:junit4] 2> 213704 T625 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 213704 T625 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 213705 T625 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 213705 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 213706 T625 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 213707 T625 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 213707 T625 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 213708 T625 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 213708 T625 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 213709 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 213713 T625 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 213714 T625 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44835/solr
[junit4:junit4] 2> 213714 T625 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 213715 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 213717 T714 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c5415ada name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 213717 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 213719 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 213725 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 213727 T716 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dd5e564 name:ZooKeeperConnection Watcher:127.0.0.1:44835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 213727 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 213730 T625 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 213927 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 213927 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57727/_/nx",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:57727__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57727__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 213931 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 213931 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 213931 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 213931 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 213931 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 213931 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 214591 T660 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:57727__%2Fnx_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 214591 T660 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:57727__%252Fnx_collection1&wt=javabin&nodeName=127.0.0.1:57727__%252Fnx&core=collection1} status=0 QTime=1001
[junit4:junit4] 2> 214733 T625 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41723__%2Fnx
[junit4:junit4] 2> 214734 T625 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41723__%2Fnx
[junit4:junit4] 2> 214736 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 214736 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 214736 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 214737 T716 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 214737 T654 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 214737 T647 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 214738 T667 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 214738 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 214738 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 214738 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 214739 T683 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 214740 T699 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 214742 T717 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 214743 T717 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 215435 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 215436 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41723/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41723__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 215436 T648 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 215437 T648 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 215442 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 215442 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 215442 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 215442 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 215442 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 215442 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 215744 T717 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/collection1
[junit4:junit4] 2> 215744 T717 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 215745 T717 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 215745 T717 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 215746 T717 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/collection1/'
[junit4:junit4] 2> 215747 T717 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/collection1/lib/README' to classloader
[junit4:junit4] 2> 215747 T717 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 215771 T717 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 215819 T717 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 215820 T717 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 215827 T717 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 216160 T717 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 216165 T717 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 216167 T717 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 216178 T717 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 216181 T717 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 216185 T717 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 216186 T717 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 216186 T717 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 216187 T717 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 216188 T717 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 216188 T717 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 216188 T717 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 216189 T717 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372111625987/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/
[junit4:junit4] 2> 216189 T717 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@be4b0b85
[junit4:junit4] 2> 216189 T717 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 216190 T717 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4
[junit4:junit4] 2> 216190 T717 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index/
[junit4:junit4] 2> 216191 T717 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 216191 T717 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index
[junit4:junit4] 2> 216192 T717 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@c89b0201 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf7e56d0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 216193 T717 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 216194 T717 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 216195 T717 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 216195 T717 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 216196 T717 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 216196 T717 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 216196 T717 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 216197 T717 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 216197 T717 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 216198 T717 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 216199 T717 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 216201 T717 oass.SolrIndexSearcher.<init> Opening Searcher@fc11a6b1 main
[junit4:junit4] 2> 216201 T717 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/tlog
[junit4:junit4] 2> 216202 T717 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 216202 T717 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 216206 T718 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fc11a6b1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 216207 T717 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 216207 T717 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41723/_/nx collection:collection1 shard:shard2
[junit4:junit4] 2> 216210 T717 oasc.ZkController.register We are http://127.0.0.1:41723/_/nx/collection1/ and leader is http://127.0.0.1:47018/_/nx/collection1/
[junit4:junit4] 2> 216210 T717 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41723/_/nx
[junit4:junit4] 2> 216210 T717 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 216210 T717 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C214 name=collection1 org.apache.solr.core.SolrCore@c1302f7f url=http://127.0.0.1:41723/_/nx/collection1 node=127.0.0.1:41723__%2Fnx C214_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:41723/_/nx, state=down, node_name=127.0.0.1:41723__%2Fnx, collection=collection1, core=collection1}
[junit4:junit4] 2> 216211 T719 C214 P41723 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 216211 T717 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 216211 T719 C214 P41723 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 216212 T719 C214 P41723 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 216212 T719 C214 P41723 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 216213 T625 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 216214 T625 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 216213 T719 C214 P41723 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 216215 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 216222 T625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 216222 T676 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:41723__%2Fnx_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 216224 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 216231 T663 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> ASYNC NEW_CORE C215 name=collection1 org.apache.solr.core.SolrCore@1bcd57e2 url=http://127.0.0.1:57727/_/nx/collection1 node=127.0.0.1:57727__%2Fnx C215_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:57727/_/nx, state=recovering, node_name=127.0.0.1:57727__%2Fnx, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 216592 T702 C215 P57727 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43587/_/nx/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 216593 T702 C215 P57727 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57727/_/nx START replicas=[http://127.0.0.1:43587/_/nx/collection1/] nUpdates=100
[junit4:junit4] 2> 216593 T702 C215 P57727 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 216594 T702 C215 P57727 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 216594 T702 C215 P57727 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 216594 T702 C215 P57727 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 216595 T702 C215 P57727 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 216595 T702 C215 P57727 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43587/_/nx/collection1/. core=collection1
[junit4:junit4] 2> 216595 T702 C215 P57727 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C216 name=collection1 org.apache.solr.core.SolrCore@fb3e4bad url=http://127.0.0.1:43587/_/nx/collection1 node=127.0.0.1:43587__%2Fnx C216_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:43587/_/nx, state=active, node_name=127.0.0.1:43587__%2Fnx, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 216600 T661 C216 P43587 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 216600 T662 C216 P43587 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 216603 T662 C216 P43587 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@17d2c538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30eb7ba0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 216603 T662 C216 P43587 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 216604 T662 C216 P43587 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@17d2c538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30eb7ba0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@17d2c538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30eb7ba0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 216605 T662 C216 P43587 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 216606 T662 C216 P43587 oass.SolrIndexSearcher.<init> Opening Searcher@7e17a7d6 realtime
[junit4:junit4] 2> 216606 T662 C216 P43587 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 216607 T662 C216 P43587 oasup.LogUpdateProcessor.finish [collection1] webapp=/_/nx path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 7
[junit4:junit4] 2> 216608 T702 C215 P57727 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 216608 T702 C215 P57727 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 216610 T661 C216 P43587 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 216610 T661 C216 P43587 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 216611 T702 C215 P57727 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 216612 T702 C215 P57727 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 216612 T702 C215 P57727 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 216614 T662 C216 P43587 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 216614 T702 C215 P57727 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 216615 T702 C215 P57727 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index.20130624110709011
[junit4:junit4] 2> 216615 T702 C215 P57727 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@ffc34339 lockFactory=org.apache.lucene.store.NativeFSLockFactory@578966c7) fullCopy=false
[junit4:junit4] 2> 216617 T660 C216 P43587 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 216619 T702 C215 P57727 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 216620 T702 C215 P57727 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 216621 T702 C215 P57727 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 216622 T702 C215 P57727 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@128778e5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60bf049a; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@128778e5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60bf049a; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 216622 T702 C215 P57727 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 216623 T702 C215 P57727 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 216623 T702 C215 P57727 oass.SolrIndexSearcher.<init> Opening Searcher@d9b26d6e main
[junit4:junit4] 2> 216624 T701 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d9b26d6e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 216625 T702 C215 P57727 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index.20130624110709011 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index.20130624110709011;done=true>>]
[junit4:junit4] 2> 216625 T702 C215 P57727 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index.20130624110709011
[junit4:junit4] 2> 216626 T702 C215 P57727 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index.20130624110709011
[junit4:junit4] 2> 216626 T702 C215 P57727 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 216626 T702 C215 P57727 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 216627 T702 C215 P57727 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 216627 T702 C215 P57727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 216628 T702 C215 P57727 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 216947 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 216948 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41723/_/nx",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:41723__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41723__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 216952 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 216952 T648 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 216952 T648 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 216955 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57727/_/nx",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:57727__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57727__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 216959 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 216960 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 216960 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 216960 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 216961 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 216960 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 217223 T676 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:41723__%2Fnx_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 217224 T676 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:41723__%252Fnx_collection1&wt=javabin&nodeName=127.0.0.1:41723__%252Fnx&core=collection1} status=0 QTime=1002
[junit4:junit4] 2> 217233 T663 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/onenodecollectioncore
[junit4:junit4] 2> 217234 T663 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 217235 T663 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 217236 T663 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 217237 T663 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 217238 T663 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 217240 T663 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 217241 T663 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/onenodecollectioncore/'
[junit4:junit4] 2> 217266 T663 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 217296 T663 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 217298 T663 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 217303 T663 oass.IndexSchema.readSchema [onenodecollectioncore] Schema name=test
[junit4:junit4] 2> 217691 T663 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 217696 T663 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 217700 T663 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 217711 T663 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 217715 T663 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 217718 T663 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 217719 T663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 217720 T663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 217720 T663 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 217721 T663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 217722 T663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 217723 T663 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 217723 T663 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372111617317/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/
[junit4:junit4] 2> 217724 T663 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@be4b0b85
[junit4:junit4] 2> 217724 T663 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 217726 T663 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection
[junit4:junit4] 2> 217726 T663 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/index/
[junit4:junit4] 2> 217727 T663 oasc.SolrCore.initIndex WARN [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 217727 T663 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/index
[junit4:junit4] 2> 217729 T663 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@b943ee28 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9a9f9a7e; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 217730 T663 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 217731 T663 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 217732 T663 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 217733 T663 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 217734 T663 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 217734 T663 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 217735 T663 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 217735 T663 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 217736 T663 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 217737 T663 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 217738 T663 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 217740 T663 oass.SolrIndexSearcher.<init> Opening Searcher@80a254a main
[junit4:junit4] 2> 217741 T663 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/tlog
[junit4:junit4] 2> 217742 T663 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 217742 T663 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 217746 T723 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@80a254a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 217749 T663 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 217750 T663 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:43587/_/nx collection:onenodecollection shard:shard1
[junit4:junit4] 2> 217751 T663 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 217757 T663 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 217759 T663 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 217760 T663 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 217761 T663 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43587/_/nx/onenodecollectioncore/
[junit4:junit4] 2> 217762 T663 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 217763 T663 oasc.SyncStrategy.syncToMe http://127.0.0.1:43587/_/nx/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 217764 T663 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43587/_/nx/onenodecollectioncore/
[junit4:junit4] 2> 217765 T663 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 218470 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 218478 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 218478 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 218478 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 218479 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 218478 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 218478 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 218522 T663 oasc.ZkController.register We are http://127.0.0.1:43587/_/nx/onenodecollectioncore/ and leader is http://127.0.0.1:43587/_/nx/onenodecollectioncore/
[junit4:junit4] 2> 218522 T663 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:43587/_/nx
[junit4:junit4] 2> 218523 T663 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 218524 T663 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 218525 T663 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 218526 T663 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection&action=CREATE&version=2&numShards=1&name=onenodecollectioncore&wt=javabin&roles=none&collection=onenodecollection} status=0 QTime=2295
[junit4:junit4] 2> 218527 T625 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 218528 T625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C217 name=collection1 org.apache.solr.core.SolrCore@c1302f7f url=http://127.0.0.1:41723/_/nx/collection1 node=127.0.0.1:41723__%2Fnx C217_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:41723/_/nx, state=recovering, node_name=127.0.0.1:41723__%2Fnx, collection=collection1, shard=shard2, core=collection1}
[junit4:junit4] 2> 219225 T719 C217 P41723 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:47018/_/nx/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 219226 T719 C217 P41723 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41723/_/nx START replicas=[http://127.0.0.1:47018/_/nx/collection1/] nUpdates=100
[junit4:junit4] 2> 219227 T719 C217 P41723 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 219227 T719 C217 P41723 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 219227 T719 C217 P41723 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 219228 T719 C217 P41723 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 219228 T719 C217 P41723 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 219228 T719 C217 P41723 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:47018/_/nx/collection1/. core=collection1
[junit4:junit4] 2> 219229 T719 C217 P41723 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C218 name=collection1 org.apache.solr.core.SolrCore@4a85ce77 url=http://127.0.0.1:47018/_/nx/collection1 node=127.0.0.1:47018__%2Fnx C218_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:47018/_/nx, state=active, node_name=127.0.0.1:47018__%2Fnx, collection=collection1, shard=shard2, core=collection1, leader=true}
[junit4:junit4] 2> 219231 T677 C218 P47018 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 219236 T679 C218 P47018 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 219238 T679 C218 P47018 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@d0aec391 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48271ad4; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> 219239 T679 C218 P47018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 219241 T679 C218 P47018 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@d0aec391 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48271ad4; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@d0aec391 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48271ad4; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 219242 T679 C218 P47018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 219243 T679 C218 P47018 oass.SolrIndexSearcher.<init> Opening Searcher@41427aa2 realtime
[junit4:junit4] 2> 219244 T679 C218 P47018 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 219245 T679 C218 P47018 oasup.LogUpdateProcessor.finish [collection1] webapp=/_/nx path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 9
[junit4:junit4] 2> 219246 T719 C217 P41723 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 219247 T719 C217 P41723 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 219248 T677 C218 P47018 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 219249 T677 C218 P47018 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 219249 T719 C217 P41723 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 219250 T719 C217 P41723 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 219250 T719 C217 P41723 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 219252 T678 C218 P47018 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 219253 T719 C217 P41723 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 219254 T719 C217 P41723 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index.20130624110711650
[junit4:junit4] 2> 219254 T719 C217 P41723 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@fba630c9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@542e8f02) fullCopy=false
[junit4:junit4] 2> 219257 T679 C218 P47018 oasc.SolrCore.execute [collection1] webapp=/_/nx path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=1
[junit4:junit4] 2> 219261 T719 C217 P41723 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 219262 T719 C217 P41723 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 219263 T719 C217 P41723 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 219264 T719 C217 P41723 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@c89b0201 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf7e56d0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@c89b0201 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf7e56d0; maxCacheMB=0.94921875 maxMergeSizeMB=0.6396484375)),segFN=segments_2,generation=2}
[junit4:junit4] 2> 219265 T719 C217 P41723 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 219265 T719 C217 P41723 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 219266 T719 C217 P41723 oass.SolrIndexSearcher.<init> Opening Searcher@38fb1e08 main
[junit4:junit4] 2> 219267 T718 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38fb1e08 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 219268 T719 C217 P41723 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index.20130624110711650 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index.20130624110711650;done=true>>]
[junit4:junit4] 2> 219268 T719 C217 P41723 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index.20130624110711650
[junit4:junit4] 2> 219268 T719 C217 P41723 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index.20130624110711650
[junit4:junit4] 2> 219269 T719 C217 P41723 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 219269 T719 C217 P41723 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 219269 T719 C217 P41723 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 219270 T719 C217 P41723 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 219272 T719 C217 P41723 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 219530 T625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 219984 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 219985 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:43587__%2Fnx_onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 219988 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41723/_/nx",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:41723__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:41723__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 219992 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 219992 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 219992 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 219992 T654 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 219992 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 219992 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 220532 T625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 220533 T625 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 220534 T625 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 220546 T625 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 220548 T727 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dafd24a name:ZooKeeperConnection Watcher:127.0.0.1:44835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 220548 T625 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89922706708299779-127.0.0.1:35055__%2Fnx-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (5)
[junit4:junit4] 1> /solr/overseer_elect/election/89922706708299782-127.0.0.1:43587__%2Fnx-n_0000000001 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89922706708299779-127.0.0.1:35055__%2Fnx-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89922706708299784-127.0.0.1:47018__%2Fnx-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89922706708299786-127.0.0.1:57727__%2Fnx-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89922706708299788-127.0.0.1:41723__%2Fnx-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89922706708299779-127.0.0.1:35055__%2Fnx_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:35055/_/nx",
[junit4:junit4] 1> "node_name":"127.0.0.1:35055__%2Fnx",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89922706708299782-127.0.0.1:43587__%2Fnx_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89922706708299786-127.0.0.1:57727__%2Fnx_collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89922706708299788-127.0.0.1:41723__%2Fnx_collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89922706708299784-127.0.0.1:47018__%2Fnx_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 1> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47018/_/nx",
[junit4:junit4] 1> "node_name":"127.0.0.1:47018__%2Fnx",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89922706708299782-127.0.0.1:43587__%2Fnx_onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 1> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (5)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:35055__%2Fnx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:43587__%2Fnx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:47018__%2Fnx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:41723__%2Fnx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:57727__%2Fnx (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:35055__%2Fnx_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:35055/_/nx",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:35055__%2Fnx",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "onenodecollection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:43587__%2Fnx_onenodecollectioncore":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 1> "roles":"none",
[junit4:junit4] 1> "collection":"onenodecollection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"onenodecollectioncore",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{
[junit4:junit4] 1> "shard1":{
[junit4:junit4] 1> "range":"80000000-ffffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:43587__%2Fnx_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:57727__%2Fnx_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:57727/_/nx",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:57727__%2Fnx",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}},
[junit4:junit4] 1> "shard2":{
[junit4:junit4] 1> "range":"0-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:47018__%2Fnx_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47018/_/nx",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:47018__%2Fnx",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:41723__%2Fnx_collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41723/_/nx",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:41723__%2Fnx",
[junit4:junit4] 2> 220603 T627 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f7839ae970004, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2>
[junit4:junit4] 2> 220607 T625 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 35055
[junit4:junit4] 2> 220608 T625 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1132056162
[junit4:junit4] 2> 221497 T648 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 221498 T648 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35055/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35055__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:35055__%2Fnx",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 221502 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 221503 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 221503 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 221503 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 221503 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 222611 T625 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 222612 T625 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 222614 T625 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@cb248bc4
[junit4:junit4] 2> 222619 T625 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 222620 T625 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 222621 T625 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 222622 T625 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 222623 T625 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 222624 T625 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 222624 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/index;done=false>>]
[junit4:junit4] 2> 222625 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data/index
[junit4:junit4] 2> 222626 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data;done=false>>]
[junit4:junit4] 2> 222627 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/control/data
[junit4:junit4] 2> 222628 T648 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89922706708299779-127.0.0.1:35055__%2Fnx-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 222629 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 222629 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 222630 T716 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 222630 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 222631 T699 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 222631 T683 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 222629 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 222633 T667 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 222635 T667 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 222636 T667 oasc.Overseer.start Overseer (id=89922706708299782-127.0.0.1:43587__%2Fnx-n_0000000001) starting
[junit4:junit4] 2> 222639 T729 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 222640 T728 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 222641 T728 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 222642 T728 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35055/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35055__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:35055__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 222646 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 222646 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 222646 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 222646 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 222650 T625 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_/nx,null}
[junit4:junit4] 2> 222702 T625 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 43587
[junit4:junit4] 2> 222703 T625 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=135591031
[junit4:junit4] 2> 223929 T647 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 223929 T647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 223930 T647 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 224149 T728 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 224150 T728 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:43587__%2Fnx_onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 224153 T728 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:43587__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 224158 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 224158 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 224158 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 224158 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 225705 T625 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 225706 T625 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 225708 T625 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 225709 T625 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@fb3e4bad
[junit4:junit4] 2> 225713 T625 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 225714 T625 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 225715 T625 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 225715 T625 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 225717 T625 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 225718 T625 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 225719 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/index;done=false>>]
[junit4:junit4] 2> 225720 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1/index
[junit4:junit4] 2> 225720 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1;done=false>>]
[junit4:junit4] 2> 225721 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty1
[junit4:junit4] 2> 225721 T625 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@8198c317
[junit4:junit4] 2> 225728 T625 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 225729 T625 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 225730 T625 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 225731 T625 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 225732 T625 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 225733 T625 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 225734 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection;done=false>>]
[junit4:junit4] 2> 225735 T625 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection
[junit4:junit4] 2> 225736 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/index;done=false>>]
[junit4:junit4] 2> 225737 T625 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372111613587/onenodecollection/index
[junit4:junit4] 2> 225738 T728 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89922706708299782-127.0.0.1:43587__%2Fnx-n_0000000001) am no longer a leader.
[junit4:junit4] 2> 225739 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 225739 T627 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f7839ae970006, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2>
[junit4:junit4] 2> 225739 T667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 225739 T716 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 225740 T683 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 225740 T667 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 225740 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 225741 T699 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 225741 T667 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 225742 T683 oasc.Overseer.start Overseer (id=89922706708299784-127.0.0.1:47018__%2Fnx-n_0000000002) starting
[junit4:junit4] 2> 225743 T699 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 225744 T699 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 225745 T699 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 225745 T699 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 225745 T699 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57727/_/nx/collection1/
[junit4:junit4] 2> 225746 T699 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 225746 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 225746 T699 oasc.SyncStrategy.syncToMe http://127.0.0.1:57727/_/nx/collection1/ has no replicas
[junit4:junit4] 2> 225746 T730 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 225746 T731 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 225746 T699 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57727/_/nx/collection1/
[junit4:junit4] 2> 225747 T683 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 225747 T699 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 225748 T730 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 225749 T730 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:43587__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 225752 T730 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43587/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:43587__%2Fnx_onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:43587__%2Fnx",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 225760 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 225760 T625 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_/nx,null}
[junit4:junit4] 2> 225760 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 225760 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 225812 T625 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47018
[junit4:junit4] 2> 225813 T625 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1763847281
[junit4:junit4] 2> 226815 T625 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 226816 T625 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 226817 T625 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4a85ce77
[junit4:junit4] 2> 226820 T625 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 226821 T625 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 226821 T625 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 226822 T625 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 226823 T625 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 226824 T625 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 226824 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2;done=false>>]
[junit4:junit4] 2> 226825 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2
[junit4:junit4] 2> 226825 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/index;done=false>>]
[junit4:junit4] 2> 226826 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty2/index
[junit4:junit4] 2> 226826 T730 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89922706708299784-127.0.0.1:47018__%2Fnx-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 226827 T683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 226828 T683 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 226828 T683 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 226828 T699 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 226828 T716 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 226830 T699 oasc.Overseer.start Overseer (id=89922706708299786-127.0.0.1:57727__%2Fnx-n_0000000003) starting
[junit4:junit4] 2> 226830 T716 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 226831 T716 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 226831 T716 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 226831 T716 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41723/_/nx/collection1/
[junit4:junit4] 2> 226832 T716 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41723/_/nx START replicas=[http://127.0.0.1:47018/_/nx/collection1/] nUpdates=100
[junit4:junit4] 2> 226832 T716 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41723/_/nx DONE. We have no versions. sync failed.
[junit4:junit4] 2> 226833 T716 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 226833 T716 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 226833 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 226833 T734 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 226834 T732 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 226834 T716 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41723/_/nx/collection1/
[junit4:junit4] 2> 226835 T716 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 226835 T699 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 226836 T732 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 226837 T732 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47018/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:47018__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47018__%2Fnx",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 226838 T625 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_/nx,null}
[junit4:junit4] 2> 226838 T716 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 226839 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 226839 T732 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47018/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:47018__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47018__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 226846 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 226846 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 226889 T625 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 57727
[junit4:junit4] 2> 226890 T625 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-2023196345
[junit4:junit4] 2> 227893 T625 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 227894 T625 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 227899 T625 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1bcd57e2
[junit4:junit4] 2> 227905 T625 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 227905 T625 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 227906 T625 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 227906 T625 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 227907 T625 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 227908 T625 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 227908 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3;done=false>>]
[junit4:junit4] 2> 227909 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3
[junit4:junit4] 2> 227909 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index;done=false>>]
[junit4:junit4] 2> 227910 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty3/index
[junit4:junit4] 2> 227910 T732 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89922706708299786-127.0.0.1:57727__%2Fnx-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 227911 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 227912 T699 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 227912 T699 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 227913 T716 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 227914 T716 oasc.Overseer.start Overseer (id=89922706708299788-127.0.0.1:41723__%2Fnx-n_0000000004) starting
[junit4:junit4] 2> 227917 T736 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 227919 T716 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 227919 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 227921 T735 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 227924 T735 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 227924 T735 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57727/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:57727__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57727__%2Fnx",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 227928 T735 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57727/_/nx",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:57727__%2Fnx_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57727__%2Fnx",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 227932 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 227933 T625 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_/nx,null}
[junit4:junit4] 2> 227984 T625 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 41723
[junit4:junit4] 2> 227985 T625 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=716364007
[junit4:junit4] 2> 228987 T625 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 228988 T625 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 228990 T625 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@c1302f7f
[junit4:junit4] 2> 228995 T625 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 228996 T625 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 228996 T625 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 228997 T625 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 228998 T625 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 228998 T625 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 228999 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index;done=false>>]
[junit4:junit4] 2> 228999 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4/index
[junit4:junit4] 2> 229000 T625 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4;done=false>>]
[junit4:junit4] 2> 229000 T625 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372111613587/jetty4
[junit4:junit4] 2> 229001 T735 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89922706708299788-127.0.0.1:41723__%2Fnx-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 229023 T625 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_/nx,null}
[junit4:junit4] 2> 229078 T625 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 229079 T625 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44835 44835
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=475CA3DF2F55AE9F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=Pacific/Samoa -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 27.9s J1 | BasicDistributedZk2Test.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:47018/_/nx/onenodecollectioncore returned non ok status:404, message:Can not find: /_/nx/onenodecollectioncore/update
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([475CA3DF2F55AE9F:C6BA2DC7580ACEA3]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
[junit4:junit4] > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
[junit4:junit4] > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
[junit4:junit4] > at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
[junit4:junit4] > at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2> 229100 T625 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 27921 T624 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 230302 T716 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 230302 T716 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 230302 T716 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {}, docValues:{}, sim=DefaultSimilarity, locale=es_HN, timezone=Pacific/Samoa
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=2,free=23857520,total=165150720
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestCopyFieldCollectionResource, CollectionsAPIDistributedZkTest, TestStressReorder, UpdateParamsTest, SimplePostToolTest, TestBinaryResponseWriter, DateMathParserTest, TestQueryTypes, FileUtilsTest, NumericFieldsTest, IndexBasedSpellCheckerTest, DirectUpdateHandlerTest, HardAutoCommitTest, StandardRequestHandlerTest, SuggesterWFSTTest, TestCSVLoader, TestSchemaSimilarityResource, DistanceFunctionTest, DocValuesMultiTest, SignatureUpdateProcessorFactoryTest, TestDistributedSearch, MultiTermTest, TestManagedSchema, StatsComponentTest, SortByFunctionTest, TestFieldResource, SpellCheckComponentTest, PathHierarchyTokenizerFactoryTest, BasicDistributedZk2Test]
[junit4:junit4] Completed on J1 in 29.20s, 1 test, 1 error <<< FAILURES!
[...truncated 752 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:379: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 error, 14 ignored (8 assumptions)
Total time: 40 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure