You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/18 19:49:07 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_21) - Build #
6195 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6195/
Java: 32bit/jdk1.7.0_21 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<620> but was:<52>
Stack Trace:
java.lang.AssertionError: expected:<620> but was:<52>
at __randomizedtesting.SeedInfo.seed([B9548118778436:815FDA996F28E40A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9597 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 659414 T3171 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /tp/au
[junit4:junit4] 2> 659417 T3171 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371577097378
[junit4:junit4] 2> 659418 T3171 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 659418 T3172 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 659518 T3171 oasc.ZkTestServer.run start zk server on port:35894
[junit4:junit4] 2> 659519 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 659862 T3178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c6f8c name:ZooKeeperConnection Watcher:127.0.0.1:35894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 659863 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 659863 T3171 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 659868 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 659870 T3180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ad5467 name:ZooKeeperConnection Watcher:127.0.0.1:35894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 659871 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 659871 T3171 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 659875 T3171 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 659879 T3171 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 659882 T3171 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 659886 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 659887 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 659893 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 659894 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 659898 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 659899 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 659903 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 659904 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 659908 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 659909 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 659914 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 659915 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 659919 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 659920 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 659923 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 659924 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 659928 T3171 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 659929 T3171 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 660007 T3171 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 660010 T3171 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60338
[junit4:junit4] 2> 660011 T3171 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 660011 T3171 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 660012 T3171 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893
[junit4:junit4] 2> 660012 T3171 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/solr.xml
[junit4:junit4] 2> 660012 T3171 oasc.CoreContainer.<init> New CoreContainer 1486713
[junit4:junit4] 2> 660013 T3171 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/'
[junit4:junit4] 2> 660013 T3171 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/'
[junit4:junit4] 2> 660067 T3171 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 660067 T3171 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 660067 T3171 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 660068 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 660068 T3171 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 660069 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 660069 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 660069 T3171 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 660070 T3171 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 660070 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 660073 T3171 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 660074 T3171 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35894/solr
[junit4:junit4] 2> 660074 T3171 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 660075 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 660076 T3191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e4af65 name:ZooKeeperConnection Watcher:127.0.0.1:35894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 660076 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 660078 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 660080 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 660082 T3193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@213a9e name:ZooKeeperConnection Watcher:127.0.0.1:35894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 660082 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 660084 T3171 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 660086 T3171 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 660088 T3171 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 660089 T3171 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60338_tp%2Fau
[junit4:junit4] 2> 660090 T3171 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60338_tp%2Fau
[junit4:junit4] 2> 660092 T3171 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 660096 T3171 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 660098 T3171 oasc.Overseer.start Overseer (id=89887676653961219-127.0.0.1:60338_tp%2Fau-n_0000000000) starting
[junit4:junit4] 2> 660100 T3171 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 660104 T3195 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 660104 T3171 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 660107 T3171 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 660110 T3171 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 660114 T3194 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 660118 T3196 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 660119 T3196 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 661617 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 661617 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60338_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60338/tp/au"}
[junit4:junit4] 2> 661618 T3194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 661618 T3194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 661623 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 662120 T3196 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/collection1
[junit4:junit4] 2> 662121 T3196 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 662122 T3196 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 662122 T3196 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 662124 T3196 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/collection1/'
[junit4:junit4] 2> 662125 T3196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/collection1/lib/README' to classloader
[junit4:junit4] 2> 662126 T3196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 662170 T3196 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 662206 T3196 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 662208 T3196 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 662212 T3196 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 662591 T3196 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 662597 T3196 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 662600 T3196 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 662610 T3196 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 662613 T3196 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 662617 T3196 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 662618 T3196 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 662618 T3196 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 662618 T3196 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 662619 T3196 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 662620 T3196 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 662620 T3196 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 662620 T3196 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371577097893/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/control/data/
[junit4:junit4] 2> 662620 T3196 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1395a3e
[junit4:junit4] 2> 662621 T3196 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 662621 T3196 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/control/data
[junit4:junit4] 2> 662621 T3196 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/control/data/index/
[junit4:junit4] 2> 662621 T3196 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371577097377/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 662622 T3196 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/control/data/index
[junit4:junit4] 2> 662623 T3196 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@755b63 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee9531),segFN=segments_1,generation=1}
[junit4:junit4] 2> 662623 T3196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 662624 T3196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 662625 T3196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 662625 T3196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 662625 T3196 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 662626 T3196 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 662626 T3196 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 662626 T3196 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 662627 T3196 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 662627 T3196 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 662629 T3196 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 662632 T3196 oass.SolrIndexSearcher.<init> Opening Searcher@1b9c456 main
[junit4:junit4] 2> 662632 T3196 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/control/data/tlog
[junit4:junit4] 2> 662632 T3196 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 662633 T3196 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 662637 T3197 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b9c456 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 662638 T3196 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 662638 T3196 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60338/tp/au collection:control_collection shard:shard1
[junit4:junit4] 2> 662638 T3196 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 662643 T3196 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 662644 T3196 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 662645 T3196 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 662645 T3196 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60338/tp/au/collection1/
[junit4:junit4] 2> 662645 T3196 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 662645 T3196 oasc.SyncStrategy.syncToMe http://127.0.0.1:60338/tp/au/collection1/ has no replicas
[junit4:junit4] 2> 662645 T3196 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60338/tp/au/collection1/
[junit4:junit4] 2> 662646 T3196 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 663128 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 663137 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 663151 T3196 oasc.ZkController.register We are http://127.0.0.1:60338/tp/au/collection1/ and leader is http://127.0.0.1:60338/tp/au/collection1/
[junit4:junit4] 2> 663151 T3196 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60338/tp/au
[junit4:junit4] 2> 663152 T3196 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 663152 T3196 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 663153 T3196 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 663155 T3196 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 663157 T3171 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 663158 T3171 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 663158 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 663163 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 663164 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 663165 T3200 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bbf722 name:ZooKeeperConnection Watcher:127.0.0.1:35894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 663165 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 663167 T3171 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 663169 T3171 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 663235 T3171 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 663237 T3171 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48395
[junit4:junit4] 2> 663237 T3171 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 663238 T3171 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 663238 T3171 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130
[junit4:junit4] 2> 663238 T3171 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/solr.xml
[junit4:junit4] 2> 663239 T3171 oasc.CoreContainer.<init> New CoreContainer 5075668
[junit4:junit4] 2> 663239 T3171 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/'
[junit4:junit4] 2> 663240 T3171 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/'
[junit4:junit4] 2> 663288 T3171 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 663289 T3171 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 663289 T3171 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 663290 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 663290 T3171 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 663290 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 663291 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 663291 T3171 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 663291 T3171 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 663292 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 663295 T3171 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 663295 T3171 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35894/solr
[junit4:junit4] 2> 663296 T3171 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 663297 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 663298 T3211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f965b name:ZooKeeperConnection Watcher:127.0.0.1:35894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 663299 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 663301 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 663304 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 663305 T3213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c7c190 name:ZooKeeperConnection Watcher:127.0.0.1:35894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 663305 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 663310 T3171 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 664314 T3171 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48395_tp%2Fau
[junit4:junit4] 2> 664316 T3171 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48395_tp%2Fau
[junit4:junit4] 2> 664319 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 664319 T3200 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 664320 T3213 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 664321 T3193 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 664329 T3214 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 664329 T3214 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 664644 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 664645 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60338_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60338/tp/au"}
[junit4:junit4] 2> 664651 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48395_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48395/tp/au"}
[junit4:junit4] 2> 664652 T3194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 664652 T3194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 664659 T3200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 664659 T3213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 664659 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 665331 T3214 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/collection1
[junit4:junit4] 2> 665332 T3214 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 665333 T3214 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 665333 T3214 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 665335 T3214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/collection1/'
[junit4:junit4] 2> 665336 T3214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/collection1/lib/README' to classloader
[junit4:junit4] 2> 665337 T3214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 665376 T3214 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 665411 T3214 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 665412 T3214 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 665416 T3214 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 665820 T3214 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 665826 T3214 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 665828 T3214 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 665839 T3214 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 665843 T3214 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 665846 T3214 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 665847 T3214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 665847 T3214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 665847 T3214 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 665848 T3214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 665849 T3214 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 665849 T3214 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 665849 T3214 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371577101130/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty1/
[junit4:junit4] 2> 665849 T3214 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1395a3e
[junit4:junit4] 2> 665850 T3214 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 665850 T3214 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty1
[junit4:junit4] 2> 665851 T3214 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty1/index/
[junit4:junit4] 2> 665851 T3214 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 665851 T3214 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty1/index
[junit4:junit4] 2> 665853 T3214 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c45e77 lockFactory=org.apache.lucene.store.NativeFSLockFactory@374961),segFN=segments_1,generation=1}
[junit4:junit4] 2> 665853 T3214 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 665855 T3214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 665855 T3214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 665855 T3214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 665856 T3214 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 665857 T3214 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 665857 T3214 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 665857 T3214 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 665858 T3214 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 665858 T3214 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 665861 T3214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 665864 T3214 oass.SolrIndexSearcher.<init> Opening Searcher@2cfae9 main
[junit4:junit4] 2> 665865 T3214 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty1/tlog
[junit4:junit4] 2> 665865 T3214 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 665866 T3214 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 665874 T3215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2cfae9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 665877 T3214 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 665877 T3214 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48395/tp/au collection:collection1 shard:shard1
[junit4:junit4] 2> 665879 T3214 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 665887 T3214 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 665889 T3214 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 665890 T3214 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 665890 T3214 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48395/tp/au/collection1/
[junit4:junit4] 2> 665891 T3214 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 665891 T3214 oasc.SyncStrategy.syncToMe http://127.0.0.1:48395/tp/au/collection1/ has no replicas
[junit4:junit4] 2> 665891 T3214 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48395/tp/au/collection1/
[junit4:junit4] 2> 665892 T3214 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 666166 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 666174 T3200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 666174 T3213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 666174 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 666198 T3214 oasc.ZkController.register We are http://127.0.0.1:48395/tp/au/collection1/ and leader is http://127.0.0.1:48395/tp/au/collection1/
[junit4:junit4] 2> 666199 T3214 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48395/tp/au
[junit4:junit4] 2> 666199 T3214 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 666200 T3214 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 666200 T3214 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 666203 T3214 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 666205 T3171 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 666206 T3171 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 666206 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 666287 T3171 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 666289 T3171 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39577
[junit4:junit4] 2> 666290 T3171 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 666290 T3171 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 666290 T3171 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173
[junit4:junit4] 2> 666291 T3171 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/solr.xml
[junit4:junit4] 2> 666291 T3171 oasc.CoreContainer.<init> New CoreContainer 15471975
[junit4:junit4] 2> 666291 T3171 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/'
[junit4:junit4] 2> 666292 T3171 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/'
[junit4:junit4] 2> 666338 T3171 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 666339 T3171 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 666339 T3171 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 666339 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 666340 T3171 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 666340 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 666340 T3171 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 666341 T3171 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 666341 T3171 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 666342 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 666344 T3171 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 666345 T3171 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35894/solr
[junit4:junit4] 2> 666345 T3171 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 666346 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 666347 T3227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aba3c name:ZooKeeperConnection Watcher:127.0.0.1:35894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 666348 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 666349 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 666351 T3171 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 666352 T3229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@708f05 name:ZooKeeperConnection Watcher:127.0.0.1:35894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 666352 T3171 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 666355 T3171 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 667358 T3171 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39577_tp%2Fau
[junit4:junit4] 2> 667360 T3171 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39577_tp%2Fau
[junit4:junit4] 2> 667363 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 667363 T3213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 667363 T3229 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 667363 T3200 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 667365 T3200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 667365 T3193 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 667366 T3213 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 667372 T3230 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 667373 T3230 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 667681 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 667682 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48395_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48395/tp/au"}
[junit4:junit4] 2> 667686 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39577_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39577/tp/au"}
[junit4:junit4] 2> 667686 T3194 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 667687 T3194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 667691 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 667691 T3213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 667691 T3229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 667691 T3200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 668374 T3230 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/collection1
[junit4:junit4] 2> 668375 T3230 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 668376 T3230 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 668377 T3230 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 668378 T3230 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/collection1/'
[junit4:junit4] 2> 668379 T3230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/collection1/lib/README' to classloader
[junit4:junit4] 2> 668380 T3230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 668421 T3230 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 668457 T3230 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 668458 T3230 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 668462 T3230 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 668836 T3230 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 668842 T3230 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 668844 T3230 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 668859 T3230 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 668864 T3230 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 668869 T3230 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 668870 T3230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 668871 T3230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 668871 T3230 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 668872 T3230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 668873 T3230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 668873 T3230 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 668874 T3230 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371577104173/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/
[junit4:junit4] 2> 668874 T3230 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1395a3e
[junit4:junit4] 2> 668875 T3230 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 668875 T3230 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2
[junit4:junit4] 2> 668876 T3230 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index/
[junit4:junit4] 2> 668876 T3230 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 668877 T3230 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index
[junit4:junit4] 2> 668878 T3230 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_1,generation=1}
[junit4:junit4] 2> 668879 T3230 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 668881 T3230 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 668882 T3230 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 668883 T3230 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 668884 T3230 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 668885 T3230 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 668885 T3230 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 668885 T3230 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 668886 T3230 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 668887 T3230 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 668891 T3230 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 668895 T3230 oass.SolrIndexSearcher.<init> Opening Searcher@608277 main
[junit4:junit4] 2> 668896 T3230 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/tlog
[junit4:junit4] 2> 668897 T3230 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 668897 T3230 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 668906 T3231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@608277 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 668909 T3230 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 668910 T3230 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39577/tp/au collection:collection1 shard:shard1
[junit4:junit4] 2> 668914 T3230 oasc.ZkController.register We are http://127.0.0.1:39577/tp/au/collection1/ and leader is http://127.0.0.1:48395/tp/au/collection1/
[junit4:junit4] 2> 668914 T3230 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39577/tp/au
[junit4:junit4] 2> 668915 T3230 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 668915 T3230 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C849 name=collection1 org.apache.solr.core.SolrCore@43aade url=http://127.0.0.1:39577/tp/au/collection1 node=127.0.0.1:39577_tp%2Fau C849_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:39577_tp%2Fau, base_url=http://127.0.0.1:39577/tp/au}
[junit4:junit4] 2> 668916 T3232 C849 P39577 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 668916 T3230 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 668917 T3232 C849 P39577 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 668918 T3232 C849 P39577 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 668918 T3232 C849 P39577 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 668918 T3171 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 668919 T3171 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 668920 T3171 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 668920 T3232 C849 P39577 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 668927 T3171 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 668928 T3209 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C850 name=collection1 org.apache.solr.core.SolrCore@f6d920 url=http://127.0.0.1:60338/tp/au/collection1 node=127.0.0.1:60338_tp%2Fau C850_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:60338_tp%2Fau, base_url=http://127.0.0.1:60338/tp/au, leader=true}
[junit4:junit4] 2> 668948 T3187 C850 P60338 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@755b63 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ee9531),segFN=segments_1,generation=1}
[junit4:junit4] 2> 668949 T3187 C850 P60338 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 668954 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[1 (1438202836452114432)]} 0 8
[junit4:junit4] 2> 668954 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10001 (1438202836452114433)]} 0 8
[junit4:junit4] 2> ASYNC NEW_CORE C851 name=collection1 org.apache.solr.core.SolrCore@4f045d url=http://127.0.0.1:48395/tp/au/collection1 node=127.0.0.1:48395_tp%2Fau C851_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:48395_tp%2Fau, base_url=http://127.0.0.1:48395/tp/au, leader=true}
[junit4:junit4] 2> 668971 T3208 C851 P48395 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c45e77 lockFactory=org.apache.lucene.store.NativeFSLockFactory@374961),segFN=segments_1,generation=1}
[junit4:junit4] 2> 668972 T3208 C851 P48395 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 668975 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10001 (1438202836476231680)]} 0 5
[junit4:junit4] 2> 668975 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[1 (1438202836476231681)]} 0 5
[junit4:junit4] 2> 668988 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10001 (-1438202836495106048)]} 0 0
[junit4:junit4] 2> 668988 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[1 (-1438202836495106049)]} 0 0
[junit4:junit4] 2> 668991 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10001 (-1438202836498251776)]} 0 0
[junit4:junit4] 2> 668991 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[1 (-1438202836498251777)]} 0 0
[junit4:junit4] 2> 668994 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10002 (1438202836500348928)]} 0 1
[junit4:junit4] 2> 668995 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[2 (1438202836501397504)]} 0 1
[junit4:junit4] 2> 668998 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10002 (1438202836504543232)]} 0 1
[junit4:junit4] 2> 668998 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[2 (1438202836504543233)]} 0 1
[junit4:junit4] 2> 669020 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10003 (1438202836527611904)]} 0 1
[junit4:junit4] 2> 669021 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[3 (1438202836528660480)]} 0 1
[junit4:junit4] 2> 669024 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10003 (1438202836531806208)]} 0 1
[junit4:junit4] 2> 669024 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[3 (1438202836532854784)]} 0 1
[junit4:junit4] 2> 669045 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10004 (1438202836553826304)]} 0 1
[junit4:junit4] 2> 669046 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[4 (1438202836554874880)]} 0 1
[junit4:junit4] 2> 669049 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10004 (1438202836558020608)]} 0 1
[junit4:junit4] 2> 669049 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[4 (1438202836558020609)]} 0 1
[junit4:junit4] 2> 669071 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10005 (1438202836581089280)]} 0 1
[junit4:junit4] 2> 669072 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[5 (1438202836582137856)]} 0 1
[junit4:junit4] 2> 669074 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10005 (1438202836584235008)]} 0 1
[junit4:junit4] 2> 669075 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[5 (1438202836585283584)]} 0 1
[junit4:junit4] 2> 669150 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10006 (1438202836663926784)]} 0 1
[junit4:junit4] 2> 669151 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[6 (1438202836664975360)]} 0 1
[junit4:junit4] 2> 669153 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10006 (1438202836667072512)]} 0 1
[junit4:junit4] 2> 669154 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[6 (1438202836668121088)]} 0 1
[junit4:junit4] 2> 669198 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 669199 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39577_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39577/tp/au"}
[junit4:junit4] 2> 669204 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 669204 T3229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 669204 T3213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 669204 T3200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 669228 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10002 (-1438202836746764288)]} 0 0
[junit4:junit4] 2> 669229 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[2 (-1438202836747812864)]} 0 0
[junit4:junit4] 2> C849_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:39577_tp%2Fau, base_url=http://127.0.0.1:39577/tp/au}
[junit4:junit4] 2> 669242 T3222 C849 P39577 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_1,generation=1}
[junit4:junit4] 2> 669243 T3222 C849 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 669244 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10002 (-1438202836749910016)]} 0 2
[junit4:junit4] 2> 669244 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[2 (-1438202836750958592)]} 0 2
[junit4:junit4] 2> 669244 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10002 (-1438202836749910016)]} 0 13
[junit4:junit4] 2> 669245 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[2 (-1438202836750958592)]} 0 14
[junit4:junit4] 2> 669248 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10007 (1438202836766687232)]} 0 1
[junit4:junit4] 2> 669248 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[7 (1438202836766687233)]} 0 1
[junit4:junit4] 2> 669254 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10007 (1438202836769832960)]} 0 1
[junit4:junit4] 2> 669255 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1438202836770881536)]} 0 1
[junit4:junit4] 2> 669255 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10007 (1438202836769832960)]} 0 5
[junit4:junit4] 2> 669256 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[7 (1438202836770881536)]} 0 5
[junit4:junit4] 2> 669284 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10008 (1438202836804435968)]} 0 1
[junit4:junit4] 2> 669285 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[8 (1438202836805484544)]} 0 1
[junit4:junit4] 2> 669290 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10008 (1438202836808630272)]} 0 1
[junit4:junit4] 2> 669291 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10008 (1438202836808630272)]} 0 4
[junit4:junit4] 2> 669291 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1438202836808630273)]} 0 1
[junit4:junit4] 2> 669292 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[8 (1438202836808630273)]} 0 5
[junit4:junit4] 2> 669301 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10009 (1438202836823310336)]} 0 1
[junit4:junit4] 2> 669303 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[9 (1438202836824358912)]} 0 1
[junit4:junit4] 2> 669307 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10009 (1438202836826456064)]} 0 1
[junit4:junit4] 2> 669308 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10009 (1438202836826456064)]} 0 4
[junit4:junit4] 2> 669308 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1438202836827504640)]} 0 1
[junit4:junit4] 2> 669309 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[9 (1438202836827504640)]} 0 4
[junit4:junit4] 2> 669347 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10010 (1438202836870496256)]} 0 1
[junit4:junit4] 2> 669348 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10 (1438202836872593408)]} 0 1
[junit4:junit4] 2> 669353 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10010 (1438202836873641984)]} 0 0
[junit4:junit4] 2> 669353 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10010 (1438202836873641984)]} 0 4
[junit4:junit4] 2> 669354 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1438202836875739136)]} 0 1
[junit4:junit4] 2> 669355 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10 (1438202836875739136)]} 0 4
[junit4:junit4] 2> 669425 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10011 (1438202836952285184)]} 0 1
[junit4:junit4] 2> 669426 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[11 (1438202836953333760)]} 0 1
[junit4:junit4] 2> 669431 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10011 (1438202836956479488)]} 0 1
[junit4:junit4] 2> 669432 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1438202836957528064)]} 0 1
[junit4:junit4] 2> 669432 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10011 (1438202836956479488)]} 0 4
[junit4:junit4] 2> 669433 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[11 (1438202836957528064)]} 0 4
[junit4:junit4] 2> 669459 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10006 (-1438202836988985344)]} 0 0
[junit4:junit4] 2> 669460 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[6 (-1438202836990033920)]} 0 0
[junit4:junit4] 2> 669463 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10006 (-1438202836992131072)]} 0 0
[junit4:junit4] 2> 669464 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[6 (-1438202836992131073)]} 0 0
[junit4:junit4] 2> 669464 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10006 (-1438202836992131072)]} 0 2
[junit4:junit4] 2> 669465 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[6 (-1438202836992131073)]} 0 3
[junit4:junit4] 2> 669468 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10012 (1438202836997373952)]} 0 1
[junit4:junit4] 2> 669468 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[12 (1438202836998422528)]} 0 0
[junit4:junit4] 2> 669473 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10012 (1438202837000519680)]} 0 1
[junit4:junit4] 2> 669474 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1438202837001568256)]} 0 1
[junit4:junit4] 2> 669474 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10012 (1438202837000519680)]} 0 4
[junit4:junit4] 2> 669475 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[12 (1438202837001568256)]} 0 4
[junit4:junit4] 2> 669540 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10007 (-1438202837073920000)]} 0 0
[junit4:junit4] 2> 669541 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[7 (-1438202837074968576)]} 0 0
[junit4:junit4] 2> 669545 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10007 (-1438202837077065728)]} 0 1
[junit4:junit4] 2> 669545 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10007 (-1438202837077065728)]} 0 2
[junit4:junit4] 2> 669546 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[7 (-1438202837078114304)]} 0 0
[junit4:junit4] 2> 669547 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[7 (-1438202837078114304)]} 0 3
[junit4:junit4] 2> 669548 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10013 (1438202837082308608)]} 0 0
[junit4:junit4] 2> 669550 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[13 (1438202837083357184)]} 0 1
[junit4:junit4] 2> 669554 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10013 (1438202837085454336)]} 0 1
[junit4:junit4] 2> 669555 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10013 (1438202837085454336)]} 0 4
[junit4:junit4] 2> 669555 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1438202837086502912)]} 0 0
[junit4:junit4] 2> 669556 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[13 (1438202837086502912)]} 0 4
[junit4:junit4] 2> 669612 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10014 (1438202837148368896)]} 0 1
[junit4:junit4] 2> 669613 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[14 (1438202837150466048)]} 0 1
[junit4:junit4] 2> 669617 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1438202837151514624)]} 0 1
[junit4:junit4] 2> 669618 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10014 (1438202837151514624)]} 0 4
[junit4:junit4] 2> 669619 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1438202837153611776)]} 0 1
[junit4:junit4] 2> 669620 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[14 (1438202837153611776)]} 0 4
[junit4:junit4] 2> 669650 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10008 (-1438202837188214784)]} 0 1
[junit4:junit4] 2> 669651 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[8 (-1438202837190311936)]} 0 0
[junit4:junit4] 2> 669654 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10008 (-1438202837191360512)]} 0 0
[junit4:junit4] 2> 669654 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10008 (-1438202837191360512)]} 0 2
[junit4:junit4] 2> 669656 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[8 (-1438202837193457664)]} 0 1
[junit4:junit4] 2> 669656 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[8 (-1438202837193457664)]} 0 2
[junit4:junit4] 2> 669657 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10015 (1438202837196603392)]} 0 0
[junit4:junit4] 2> 669659 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[15 (1438202837197651968)]} 0 1
[junit4:junit4] 2> 669661 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1438202837198700544)]} 0 0
[junit4:junit4] 2> 669661 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10015 (1438202837198700544)]} 0 2
[junit4:junit4] 2> 669663 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1438202837200797696)]} 0 1
[junit4:junit4] 2> 669663 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[15 (1438202837200797696)]} 0 2
[junit4:junit4] 2> 669703 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10016 (1438202837243789312)]} 0 1
[junit4:junit4] 2> 669704 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[16 (1438202837245886464)]} 0 0
[junit4:junit4] 2> 669707 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1438202837246935040)]} 0 1
[junit4:junit4] 2> 669708 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10016 (1438202837246935040)]} 0 4
[junit4:junit4] 2> 669709 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1438202837247983616)]} 0 1
[junit4:junit4] 2> 669709 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[16 (1438202837247983616)]} 0 3
[junit4:junit4] 2> 669786 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10010 (-1438202837331869696)]} 0 0
[junit4:junit4] 2> 669788 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10 (-1438202837333966848)]} 0 0
[junit4:junit4] 2> 669791 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10010 (-1438202837335015424)]} 0 0
[junit4:junit4] 2> 669791 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10010 (-1438202837335015424)]} 0 2
[junit4:junit4] 2> 669792 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10 (-1438202837336064000)]} 0 0
[junit4:junit4] 2> 669793 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10 (-1438202837336064000)]} 0 3
[junit4:junit4] 2> 669794 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10017 (1438202837339209728)]} 0 1
[junit4:junit4] 2> 669796 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[17 (1438202837341306880)]} 0 1
[junit4:junit4] 2> 669800 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1438202837343404032)]} 0 1
[junit4:junit4] 2> 669801 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10017 (1438202837343404032)]} 0 5
[junit4:junit4] 2> 669801 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1438202837344452608)]} 0 1
[junit4:junit4] 2> 669801 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[17 (1438202837344452608)]} 0 3
[junit4:junit4] 2> 669853 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10018 (1438202837401075712)]} 0 1
[junit4:junit4] 2> 669853 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[18 (1438202837402124288)]} 0 0
[junit4:junit4] 2> 669858 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1438202837404221440)]} 0 0
[junit4:junit4] 2> 669858 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1438202837405270016)]} 0 0
[junit4:junit4] 2> 669859 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10018 (1438202837404221440)]} 0 4
[junit4:junit4] 2> 669859 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[18 (1438202837405270016)]} 0 3
[junit4:junit4] 2> 669897 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10012 (-1438202837448261632)]} 0 0
[junit4:junit4] 2> 669898 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[12 (-1438202837449310208)]} 0 1
[junit4:junit4] 2> 669902 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10012 (-1438202837451407360)]} 0 0
[junit4:junit4] 2> 669902 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[12 (-1438202837451407361)]} 0 0
[junit4:junit4] 2> 669903 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10012 (-1438202837451407360)]} 0 3
[junit4:junit4] 2> 669903 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[12 (-1438202837451407361)]} 0 3
[junit4:junit4] 2> 669907 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10019 (1438202837457698816)]} 0 0
[junit4:junit4] 2> 669907 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[19 (1438202837457698817)]} 0 1
[junit4:junit4] 2> 669913 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1438202837460844544)]} 0 2
[junit4:junit4] 2> 669913 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1438202837461893120)]} 0 1
[junit4:junit4] 2> 669914 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10019 (1438202837460844544)]} 0 5
[junit4:junit4] 2> 669915 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[19 (1438202837461893120)]} 0 6
[junit4:junit4] 2> 669929 T3209 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 669929 T3209 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:39577_tp%252Fau&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 669938 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10020 (1438202837491253248)]} 0 0
[junit4:junit4] 2> 669939 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[20 (1438202837491253249)]} 0 1
[junit4:junit4] 2> 669942 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1438202837493350400)]} 0 1
[junit4:junit4] 2> 669942 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10020 (1438202837493350400)]} 0 2
[junit4:junit4] 2> 669942 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1438202837493350401)]} 0 0
[junit4:junit4] 2> 669943 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[20 (1438202837493350401)]} 0 3
[junit4:junit4] 2> 669965 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10015 (-1438202837519564800)]} 0 0
[junit4:junit4] 2> 669966 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[15 (-1438202837520613376)]} 0 0
[junit4:junit4] 2> 669968 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10015 (-1438202837521661952)]} 0 0
[junit4:junit4] 2> 669969 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[15 (-1438202837521661953)]} 0 0
[junit4:junit4] 2> 669969 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10015 (-1438202837521661952)]} 0 2
[junit4:junit4] 2> 669969 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[15 (-1438202837521661953)]} 0 2
[junit4:junit4] 2> 669971 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10021 (1438202837525856256)]} 0 0
[junit4:junit4] 2> 669972 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[21 (1438202837525856257)]} 0 1
[junit4:junit4] 2> 669975 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1438202837527953408)]} 0 0
[junit4:junit4] 2> 669975 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1438202837527953409)]} 0 0
[junit4:junit4] 2> 669975 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10021 (1438202837527953408)]} 0 2
[junit4:junit4] 2> 669976 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[21 (1438202837527953409)]} 0 3
[junit4:junit4] 2> 670077 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10017 (-1438202837637005312)]} 0 0
[junit4:junit4] 2> 670077 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[17 (-1438202837637005313)]} 0 0
[junit4:junit4] 2> 670080 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10017 (-1438202837639102464)]} 0 0
[junit4:junit4] 2> 670080 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[17 (-1438202837639102465)]} 0 0
[junit4:junit4] 2> 670081 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10017 (-1438202837639102464)]} 0 2
[junit4:junit4] 2> 670081 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[17 (-1438202837639102465)]} 0 2
[junit4:junit4] 2> 670083 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10022 (1438202837643296768)]} 0 0
[junit4:junit4] 2> 670083 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[22 (1438202837643296769)]} 0 0
[junit4:junit4] 2> 670087 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1438202837645393920)]} 0 0
[junit4:junit4] 2> 670087 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1438202837645393921)]} 0 0
[junit4:junit4] 2> 670087 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10022 (1438202837645393920)]} 0 2
[junit4:junit4] 2> 670088 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[22 (1438202837645393921)]} 0 3
[junit4:junit4] 2> 670171 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10018 (-1438202837735571456)]} 0 0
[junit4:junit4] 2> 670171 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[18 (-1438202837735571457)]} 0 0
[junit4:junit4] 2> 670176 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1438202837738717184)]} 0 0
[junit4:junit4] 2> 670176 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1438202837738717185)]} 0 0
[junit4:junit4] 2> 670177 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10018 (-1438202837738717184)]} 0 3
[junit4:junit4] 2> 670177 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[18 (-1438202837738717185)]} 0 3
[junit4:junit4] 2> 670180 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10023 (1438202837743960064)]} 0 1
[junit4:junit4] 2> 670181 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[23 (1438202837745008640)]} 0 1
[junit4:junit4] 2> 670185 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1438202837747105792)]} 0 1
[junit4:junit4] 2> 670185 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1438202837748154368)]} 0 1
[junit4:junit4] 2> 670185 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10023 (1438202837747105792)]} 0 3
[junit4:junit4] 2> 670185 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[23 (1438202837748154368)]} 0 3
[junit4:junit4] 2> 670274 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10024 (1438202837842526208)]} 0 1
[junit4:junit4] 2> 670274 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[24 (1438202837842526209)]} 0 1
[junit4:junit4] 2> 670278 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1438202837845671936)]} 0 0
[junit4:junit4] 2> 670278 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1438202837845671937)]} 0 0
[junit4:junit4] 2> 670278 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10024 (1438202837845671936)]} 0 2
[junit4:junit4] 2> 670279 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[24 (1438202837845671937)]} 0 3
[junit4:junit4] 2> 670332 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10020 (-1438202837904392192)]} 0 0
[junit4:junit4] 2> 670332 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[20 (-1438202837904392193)]} 0 0
[junit4:junit4] 2> 670335 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10020 (-1438202837906489344)]} 0 0
[junit4:junit4] 2> 670335 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[20 (-1438202837906489345)]} 0 0
[junit4:junit4] 2> 670336 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10020 (-1438202837906489344)]} 0 2
[junit4:junit4] 2> 670336 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[20 (-1438202837906489345)]} 0 2
[junit4:junit4] 2> 670338 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10025 (1438202837910683648)]} 0 1
[junit4:junit4] 2> 670338 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[25 (1438202837910683649)]} 0 0
[junit4:junit4] 2> 670342 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1438202837912780800)]} 0 1
[junit4:junit4] 2> 670342 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1438202837912780801)]} 0 0
[junit4:junit4] 2> 670342 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10025 (1438202837912780800)]} 0 2
[junit4:junit4] 2> 670343 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[25 (1438202837912780801)]} 0 3
[junit4:junit4] 2> 670445 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10026 (1438202838021832704)]} 0 1
[junit4:junit4] 2> 670445 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[26 (1438202838021832705)]} 0 1
[junit4:junit4] 2> 670451 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1438202838024978432)]} 0 1
[junit4:junit4] 2> 670451 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1438202838026027008)]} 0 0
[junit4:junit4] 2> 670452 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10026 (1438202838024978432)]} 0 5
[junit4:junit4] 2> 670452 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[26 (1438202838026027008)]} 0 4
[junit4:junit4] 2> 670468 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10027 (1438202838045949952)]} 0 1
[junit4:junit4] 2> 670468 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[27 (1438202838046998528)]} 0 1
[junit4:junit4] 2> 670474 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1438202838049095680)]} 0 1
[junit4:junit4] 2> 670475 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1438202838050144256)]} 0 1
[junit4:junit4] 2> 670475 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10027 (1438202838049095680)]} 0 5
[junit4:junit4] 2> 670475 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[27 (1438202838050144256)]} 0 4
[junit4:junit4] 2> 670502 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10028 (1438202838081601536)]} 0 1
[junit4:junit4] 2> 670503 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[28 (1438202838082650112)]} 0 1
[junit4:junit4] 2> 670508 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1438202838085795840)]} 0 1
[junit4:junit4] 2> 670508 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1438202838085795841)]} 0 1
[junit4:junit4] 2> 670509 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10028 (1438202838085795840)]} 0 5
[junit4:junit4] 2> 670509 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[28 (1438202838085795841)]} 0 4
[junit4:junit4] 2> 670572 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10021 (-1438202838156050432)]} 0 0
[junit4:junit4] 2> 670572 T3187 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[21 (-1438202838156050433)]} 0 0
[junit4:junit4] 2> 670576 T3225 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10021 (-1438202838158147584)]} 0 0
[junit4:junit4] 2> 670577 T3223 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[21 (-1438202838159196160)]} 0 0
[junit4:junit4] 2> 670577 T3206 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10021 (-1438202838158147584)]} 0 3
[junit4:junit4] 2> 670578 T3208 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[21 (-1438202838159196160)]} 0 3
[junit4:junit4] 2> 670581 T3189 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10029 (1438202838164439040)]} 0 1
[junit4:junit4] 2> 670582 T3186 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[29 (1438202838165487616)]} 0 1
[junit4:junit4] 2> 670587 T3224 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1438202838168633344)]} 0 1
[junit4:junit4] 2> 670588 T3222 C849 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1438202838168633345)]} 0 1
[junit4:junit4] 2> 670588 T3207 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10029 (1438202838168633344)]} 0 4
[junit4:junit4] 2> 670589 T3209 C851 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[29 (1438202838168633345)]} 0 5
[junit4:junit4] 2> 670690 T3188 C850 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10022 (-1438202838279782400)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C852 name=collection1 org.apache.solr.core.SolrCore@f6d920 url=http://127.0.0.1:60338/tp/au/collection1 node=127.0.0.1:60338_tp%2Fau C852_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:60338_tp%2Fau, base_url=http://127.0.0.1:60338/tp/au, leader=true}
[junit4:junit4] 2> 670690 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[22 (-1438202838279782401)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C853 name=collection1 org.apache.solr.core.SolrCore@43aade url=http://127.0.0.1:39577/tp/au/collection1 node=127.0.0.1:39577_tp%2Fau C853_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:39577_tp%2Fau, base_url=http://127.0.0.1:39577/tp/au}
[junit4:junit4] 2> 670701 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1438202838289219584)]} 0 0
[junit4:junit4] 2> 670702 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1438202838290268160)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C854 name=collection1 org.apache.solr.core.SolrCore@4f045d url=http://127.0.0.1:48395/tp/au/collection1 node=127.0.0.1:48395_tp%2Fau C854_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:48395_tp%2Fau, base_url=http://127.0.0.1:48395/tp/au, leader=true}
[junit4:junit4] 2> 670702 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10022 (-1438202838289219584)]} 0 3
[junit4:junit4] 2> 670703 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[22 (-1438202838290268160)]} 0 3
[junit4:junit4] 2> 670707 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10030 (1438202838296559616)]} 0 1
[junit4:junit4] 2> 670707 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[30 (1438202838296559617)]} 0 1
[junit4:junit4] 2> 670712 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1438202838299705344)]} 0 0
[junit4:junit4] 2> 670713 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1438202838300753920)]} 0 1
[junit4:junit4] 2> 670713 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10030 (1438202838299705344)]} 0 4
[junit4:junit4] 2> 670714 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[30 (1438202838300753920)]} 0 4
[junit4:junit4] 2> 670814 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10031 (1438202838409805824)]} 0 1
[junit4:junit4] 2> 670815 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[31 (1438202838409805825)]} 0 1
[junit4:junit4] 2> 670820 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1438202838412951552)]} 0 0
[junit4:junit4] 2> 670821 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1438202838414000128)]} 0 1
[junit4:junit4] 2> 670821 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10031 (1438202838412951552)]} 0 4
[junit4:junit4] 2> 670822 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[31 (1438202838414000128)]} 0 4
[junit4:junit4] 2> 670850 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10032 (1438202838446505984)]} 0 1
[junit4:junit4] 2> 670851 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[32 (1438202838447554560)]} 0 1
[junit4:junit4] 2> 670856 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1438202838449651712)]} 0 1
[junit4:junit4] 2> 670856 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1438202838450700288)]} 0 1
[junit4:junit4] 2> 670856 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10032 (1438202838449651712)]} 0 4
[junit4:junit4] 2> 670857 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[32 (1438202838450700288)]} 0 4
[junit4:junit4] 2> 670891 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10033 (1438202838490546176)]} 0 1
[junit4:junit4] 2> 670892 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[33 (1438202838490546177)]} 0 1
[junit4:junit4] 2> 670897 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1438202838493691904)]} 0 0
[junit4:junit4] 2> 670898 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1438202838494740480)]} 0 1
[junit4:junit4] 2> 670898 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10033 (1438202838493691904)]} 0 4
[junit4:junit4] 2> 670899 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[33 (1438202838494740480)]} 0 4
[junit4:junit4] 2> 670911 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10034 (1438202838511517696)]} 0 1
[junit4:junit4] 2> 670912 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[34 (1438202838511517697)]} 0 1
[junit4:junit4] 2> 670918 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1438202838514663424)]} 0 1
[junit4:junit4] 2> 670918 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1438202838515712000)]} 0 1
[junit4:junit4] 2> 670918 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10034 (1438202838514663424)]} 0 4
[junit4:junit4] 2> 670919 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[34 (1438202838515712000)]} 0 4
[junit4:junit4] 2> 670931 T3171 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 39577
[junit4:junit4] 2> 670932 T3171 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=15471975
[junit4:junit4] 2> 670999 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10035 (1438202838602743808)]} 0 1
[junit4:junit4] 2> 671000 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[35 (1438202838603792384)]} 0 1
[junit4:junit4] 2> 671005 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1438202838606938112)]} 0 1
[junit4:junit4] 2> 671006 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1438202838606938113)]} 0 1
[junit4:junit4] 2> 671006 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10035 (1438202838606938112)]} 0 5
[junit4:junit4] 2> 671007 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[35 (1438202838606938113)]} 0 5
[junit4:junit4] 2> 671086 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10025 (-1438202838695018496)]} 0 0
[junit4:junit4] 2> 671087 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[25 (-1438202838696067072)]} 0 0
[junit4:junit4] 2> 671091 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10025 (-1438202838698164224)]} 0 0
[junit4:junit4] 2> 671092 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[25 (-1438202838699212800)]} 0 0
[junit4:junit4] 2> 671092 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10025 (-1438202838698164224)]} 0 3
[junit4:junit4] 2> 671093 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[25 (-1438202838699212800)]} 0 3
[junit4:junit4] 2> 671096 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10036 (1438202838704455680)]} 0 1
[junit4:junit4] 2> 671096 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[36 (1438202838705504256)]} 0 0
[junit4:junit4] 2> 671102 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1438202838708649984)]} 0 1
[junit4:junit4] 2> 671102 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1438202838708649985)]} 0 1
[junit4:junit4] 2> 671103 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10036 (1438202838708649984)]} 0 4
[junit4:junit4] 2> 671103 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[36 (1438202838708649985)]} 0 4
[junit4:junit4] 2> 671194 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10037 (1438202838807216128)]} 0 1
[junit4:junit4] 2> 671194 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[37 (1438202838807216129)]} 0 1
[junit4:junit4] 2> 671200 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1438202838810361856)]} 0 1
[junit4:junit4] 2> 671200 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1438202838811410432)]} 0 1
[junit4:junit4] 2> 671200 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10037 (1438202838810361856)]} 0 4
[junit4:junit4] 2> 671201 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[37 (1438202838811410432)]} 0 4
[junit4:junit4] 2> 671230 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10026 (-1438202838846013440)]} 0 0
[junit4:junit4] 2> 671230 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[26 (-1438202838846013441)]} 0 0
[junit4:junit4] 2> 671234 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1438202838848110592)]} 0 0
[junit4:junit4] 2> 671235 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1438202838849159168)]} 0 1
[junit4:junit4] 2> 671235 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10026 (-1438202838848110592)]} 0 3
[junit4:junit4] 2> 671235 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[26 (-1438202838849159168)]} 0 2
[junit4:junit4] 2> 671238 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10038 (1438202838854402048)]} 0 0
[junit4:junit4] 2> 671239 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[38 (1438202838854402049)]} 0 1
[junit4:junit4] 2> 671244 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1438202838857547776)]} 0 1
[junit4:junit4] 2> 671244 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1438202838857547777)]} 0 0
[junit4:junit4] 2> 671245 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10038 (1438202838857547776)]} 0 4
[junit4:junit4] 2> 671245 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[38 (1438202838857547777)]} 0 4
[junit4:junit4] 2> 671300 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10027 (-1438202838919413760)]} 0 0
[junit4:junit4] 2> 671300 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[27 (-1438202838919413761)]} 0 0
[junit4:junit4] 2> 671304 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10027 (-1438202838921510912)]} 0 0
[junit4:junit4] 2> 671305 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[27 (-1438202838922559488)]} 0 1
[junit4:junit4] 2> 671305 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10027 (-1438202838921510912)]} 0 3
[junit4:junit4] 2> 671306 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[27 (-1438202838922559488)]} 0 3
[junit4:junit4] 2> 671309 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10039 (1438202838927802368)]} 0 1
[junit4:junit4] 2> 671309 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[39 (1438202838928850944)]} 0 0
[junit4:junit4] 2> 671314 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10039 (1438202838930948096)]} 0 1
[junit4:junit4] 2> 671315 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10039 (1438202838930948096)]} 0 4
[junit4:junit4] 2> 671315 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1438202838931996672)]} 0 1
[junit4:junit4] 2> 671316 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[39 (1438202838931996672)]} 0 4
[junit4:junit4] 2> 671352 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10040 (1438202838973939712)]} 0 1
[junit4:junit4] 2> 671354 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[40 (1438202838974988288)]} 0 1
[junit4:junit4] 2> 671358 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10040 (1438202838977085440)]} 0 1
[junit4:junit4] 2> 671359 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10040 (1438202838977085440)]} 0 4
[junit4:junit4] 2> 671361 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1438202838979182592)]} 0 1
[junit4:junit4] 2> 671362 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[40 (1438202838979182592)]} 0 5
[junit4:junit4] 2> 671402 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10029 (-1438202839025319936)]} 0 0
[junit4:junit4] 2> 671405 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[29 (-1438202839029514240)]} 0 0
[junit4:junit4] 2> 671406 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10029 (-1438202839028465664)]} 0 0
[junit4:junit4] 2> 671407 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10029 (-1438202839028465664)]} 0 3
[junit4:junit4] 2> 671410 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[29 (-1438202839032659968)]} 0 1
[junit4:junit4] 2> 671410 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[29 (-1438202839032659968)]} 0 2
[junit4:junit4] 2> 671411 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10041 (1438202839034757120)]} 0 1
[junit4:junit4] 2> 671414 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[41 (1438202839037902848)]} 0 1
[junit4:junit4] 2> 671416 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10041 (1438202839037902848)]} 0 0
[junit4:junit4] 2> 671417 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10041 (1438202839037902848)]} 0 4
[junit4:junit4] 2> 671419 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1438202839041048576)]} 0 1
[junit4:junit4] 2> 671420 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[41 (1438202839041048576)]} 0 4
[junit4:junit4] 2> 671464 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10042 (1438202839090331648)]} 0 1
[junit4:junit4] 2> 671466 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[42 (1438202839093477376)]} 0 1
[junit4:junit4] 2> 671469 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10042 (1438202839093477376)]} 0 0
[junit4:junit4] 2> 671470 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10042 (1438202839093477376)]} 0 4
[junit4:junit4] 2> 671472 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1438202839096623104)]} 0 1
[junit4:junit4] 2> 671472 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[42 (1438202839096623104)]} 0 3
[junit4:junit4] 2> 671552 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10032 (-1438202839183654912)]} 0 0
[junit4:junit4] 2> 671554 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[32 (-1438202839185752064)]} 0 0
[junit4:junit4] 2> 671556 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10032 (-1438202839185752064)]} 0 0
[junit4:junit4] 2> 671556 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10032 (-1438202839185752064)]} 0 2
[junit4:junit4] 2> 671557 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[32 (-1438202839186800640)]} 0 0
[junit4:junit4] 2> 671557 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[32 (-1438202839186800640)]} 0 2
[junit4:junit4] 2> 671559 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10043 (1438202839189946368)]} 0 1
[junit4:junit4] 2> 671559 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[43 (1438202839190994944)]} 0 0
[junit4:junit4] 2> 671564 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1438202839193092096)]} 0 1
[junit4:junit4] 2> 671564 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10043 (1438202839193092097)]} 0 1
[junit4:junit4] 2> 671566 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[43 (1438202839193092096)]} 0 5
[junit4:junit4] 2> 671566 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10043 (1438202839193092097)]} 0 5
[junit4:junit4] 2> 671651 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[34 (-1438202839287463936)]} 0 0
[junit4:junit4] 2> 671651 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10034 (-1438202839287463937)]} 0 0
[junit4:junit4] 2> 671655 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[34 (-1438202839289561088)]} 0 1
[junit4:junit4] 2> 671655 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10034 (-1438202839289561089)]} 0 0
[junit4:junit4] 2> 671655 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[34 (-1438202839289561088)]} 0 2
[junit4:junit4] 2> 671655 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10034 (-1438202839289561089)]} 0 2
[junit4:junit4] 2> 671657 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[44 (1438202839293755392)]} 0 0
[junit4:junit4] 2> 671657 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10044 (1438202839293755393)]} 0 0
[junit4:junit4] 2> 671661 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1438202839295852544)]} 0 0
[junit4:junit4] 2> 671662 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10044 (1438202839295852545)]} 0 1
[junit4:junit4] 2> 671662 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[44 (1438202839295852544)]} 0 3
[junit4:junit4] 2> 671662 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10044 (1438202839295852545)]} 0 3
[junit4:junit4] 2> 671753 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[35 (-1438202839394418688)]} 0 0
[junit4:junit4] 2> 671753 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10035 (-1438202839394418689)]} 0 0
[junit4:junit4] 2> 671756 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[35 (-1438202839396515840)]} 0 0
[junit4:junit4] 2> 671757 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10035 (-1438202839396515841)]} 0 0
[junit4:junit4] 2> 671757 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[35 (-1438202839396515840)]} 0 2
[junit4:junit4] 2> 671757 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10035 (-1438202839396515841)]} 0 2
[junit4:junit4] 2> 671759 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[45 (1438202839400710144)]} 0 0
[junit4:junit4] 2> 671759 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10045 (1438202839400710145)]} 0 0
[junit4:junit4] 2> 671763 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1438202839402807296)]} 0 0
[junit4:junit4] 2> 671763 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10045 (1438202839402807297)]} 0 0
[junit4:junit4] 2> 671763 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[45 (1438202839402807296)]} 0 2
[junit4:junit4] 2> 671764 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10045 (1438202839402807297)]} 0 3
[junit4:junit4] 2> 671768 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[38 (-1438202839410147328)]} 0 0
[junit4:junit4] 2> 671769 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10038 (-1438202839411195904)]} 0 0
[junit4:junit4] 2> 671772 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[38 (-1438202839412244480)]} 0 0
[junit4:junit4] 2> 671772 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10038 (-1438202839413293056)]} 0 0
[junit4:junit4] 2> 671772 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[38 (-1438202839412244480)]} 0 2
[junit4:junit4] 2> 671773 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10038 (-1438202839413293056)]} 0 3
[junit4:junit4] 2> 671774 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[46 (1438202839416438784)]} 0 0
[junit4:junit4] 2> 671775 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10046 (1438202839417487360)]} 0 0
[junit4:junit4] 2> 671778 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1438202839418535936)]} 0 0
[junit4:junit4] 2> 671778 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10046 (1438202839419584512)]} 0 0
[junit4:junit4] 2> 671779 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[46 (1438202839418535936)]} 0 3
[junit4:junit4] 2> 671779 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10046 (1438202839419584512)]} 0 2
[junit4:junit4] 2> 671827 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[40 (-1438202839472013312)]} 0 0
[junit4:junit4] 2> 671827 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10040 (-1438202839472013313)]} 0 0
[junit4:junit4] 2> 671830 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[40 (-1438202839473061888)]} 0 0
[junit4:junit4] 2> 671830 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10040 (-1438202839474110464)]} 0 0
[junit4:junit4] 2> 671830 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[40 (-1438202839473061888)]} 0 2
[junit4:junit4] 2> 671831 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10040 (-1438202839474110464)]} 0 2
[junit4:junit4] 2> 671832 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[47 (1438202839477256192)]} 0 0
[junit4:junit4] 2> 671833 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10047 (1438202839477256193)]} 0 1
[junit4:junit4] 2> 671836 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1438202839479353344)]} 0 0
[junit4:junit4] 2> 671837 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10047 (1438202839479353345)]} 0 1
[junit4:junit4] 2> 671837 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[47 (1438202839479353344)]} 0 3
[junit4:junit4] 2> 671837 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10047 (1438202839479353345)]} 0 3
[junit4:junit4] 2> 671883 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[42 (-1438202839530733568)]} 0 0
[junit4:junit4] 2> 671883 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10042 (-1438202839530733569)]} 0 0
[junit4:junit4] 2> 671886 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[42 (-1438202839532830720)]} 0 0
[junit4:junit4] 2> 671886 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10042 (-1438202839532830721)]} 0 0
[junit4:junit4] 2> 671887 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[42 (-1438202839532830720)]} 0 2
[junit4:junit4] 2> 671887 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10042 (-1438202839532830721)]} 0 2
[junit4:junit4] 2> 671889 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[48 (1438202839537025024)]} 0 0
[junit4:junit4] 2> 671889 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10048 (1438202839537025025)]} 0 0
[junit4:junit4] 2> 671893 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1438202839539122176)]} 0 1
[junit4:junit4] 2> 671893 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10048 (1438202839539122177)]} 0 0
[junit4:junit4] 2> 671893 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[48 (1438202839539122176)]} 0 2
[junit4:junit4] 2> 671894 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10048 (1438202839539122177)]} 0 3
[junit4:junit4] 2> 671913 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[43 (-1438202839562190848)]} 0 0
[junit4:junit4] 2> 671914 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10043 (-1438202839563239424)]} 0 0
[junit4:junit4] 2> 671916 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[43 (-1438202839564288000)]} 0 0
[junit4:junit4] 2> 671917 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10043 (-1438202839564288001)]} 0 0
[junit4:junit4] 2> 671917 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[43 (-1438202839564288000)]} 0 2
[junit4:junit4] 2> 671917 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10043 (-1438202839564288001)]} 0 2
[junit4:junit4] 2> 671919 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[49 (1438202839568482304)]} 0 0
[junit4:junit4] 2> 671919 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10049 (1438202839568482305)]} 0 0
[junit4:junit4] 2> 671923 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1438202839570579456)]} 0 0
[junit4:junit4] 2> 671923 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10049 (1438202839570579457)]} 0 0
[junit4:junit4] 2> 671923 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[49 (1438202839570579456)]} 0 2
[junit4:junit4] 2> 671924 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10049 (1438202839570579457)]} 0 3
[junit4:junit4] 2> 671930 T3232 C853 P39577 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:48395/tp/au/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 671930 T3232 C853 P39577 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39577/tp/au START replicas=[http://127.0.0.1:48395/tp/au/collection1/] nUpdates=100
[junit4:junit4] 2> 671931 T3208 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 671931 T3232 C853 P39577 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 671932 T3232 C853 P39577 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 671932 T3232 C853 P39577 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 671932 T3232 C853 P39577 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 671932 T3232 C853 P39577 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 671932 T3232 C853 P39577 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:48395/tp/au/collection1/. core=collection1
[junit4:junit4] 2> 671932 T3232 C853 P39577 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 671935 T3206 C854 P48395 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 671939 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[50 (1438202839589453824)]} 0 1
[junit4:junit4] 2> 671939 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10050 (1438202839589453825)]} 0 0
[junit4:junit4] 2> 671943 T3206 C854 P48395 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c45e77 lockFactory=org.apache.lucene.store.NativeFSLockFactory@374961),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c45e77 lockFactory=org.apache.lucene.store.NativeFSLockFactory@374961),segFN=segments_2,generation=2}
[junit4:junit4] 2> 671943 T3206 C854 P48395 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 671943 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10050 (1438202839591550977)]} 0 0
[junit4:junit4] 2> 671943 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1438202839591550976)]} 0 0
[junit4:junit4] 2> 671944 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10050 (1438202839591550977)]} 0 3
[junit4:junit4] 2> 671944 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[50 (1438202839591550976)]} 0 3
[junit4:junit4] 2> 671947 T3206 C854 P48395 oass.SolrIndexSearcher.<init> Opening Searcher@18f5cb8 realtime
[junit4:junit4] 2> 671947 T3206 C854 P48395 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 671947 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> 671948 T3232 C853 P39577 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 671948 T3232 C853 P39577 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 671949 T3209 C854 P48395 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 671950 T3209 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 671950 T3232 C853 P39577 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 671950 T3232 C853 P39577 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 671950 T3232 C853 P39577 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 671952 T3207 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 671952 T3232 C853 P39577 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 671953 T3232 C853 P39577 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index.20130618193829913
[junit4:junit4] 2> 671953 T3232 C853 P39577 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@190b5e lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c248d9) fullCopy=false
[junit4:junit4] 2> 671955 T3206 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 671958 T3209 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 671960 T3207 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 671962 T3208 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 671965 T3206 C854 P48395 oasc.SolrCore.execute [collection1] webapp=/tp/au path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 671965 T3232 C853 P39577 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 671966 T3232 C853 P39577 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 671966 T3232 C853 P39577 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 671966 T3232 C853 P39577 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 671972 T3232 C853 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_2,generation=2}
[junit4:junit4] 2> 671972 T3232 C853 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 671974 T3232 C853 P39577 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_2,generation=2}
[junit4:junit4] 2> 671974 T3232 C853 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 671974 T3232 C853 P39577 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 671975 T3232 C853 P39577 oass.SolrIndexSearcher.<init> Opening Searcher@ac8820 main
[junit4:junit4] 2> 671976 T3231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ac8820 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c86/42)}
[junit4:junit4] 2> 671977 T3232 C853 P39577 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index.20130618193829913 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index.20130618193829913;done=true>>]
[junit4:junit4] 2> 671977 T3232 C853 P39577 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index.20130618193829913
[junit4:junit4] 2> 671977 T3232 C853 P39577 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index.20130618193829913
[junit4:junit4] 2> 671979 T3232 C853 P39577 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 671979 T3240 C853 P39577 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=12563
[junit4:junit4] 2> 671982 T3240 C853 P39577 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 671983 T3240 C853 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@26c9f lockFactory=org.apache.lucene.store.NativeFSLockFactory@9270d9),segFN=segments_3,generation=3}
[junit4:junit4] 2> 671984 T3240 C853 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 671985 T3240 C853 P39577 oass.SolrIndexSearcher.<init> Opening Searcher@d9ac58 main
[junit4:junit4] 2> 671985 T3240 C853 P39577 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 671986 T3231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d9ac58 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c86/42 _1(5.0):c2)}
[junit4:junit4] 2> 671986 T3240 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] {add=[10050 (1438202839591550977), 50 (1438202839591550976)]} 0 7
[junit4:junit4] 2> 671986 T3240 C853 P39577 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0 deleteByQuery=0 errors=0 positionOfStart=12563}
[junit4:junit4] 2> 671986 T3232 C853 P39577 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 671987 T3232 C853 P39577 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 671987 T3232 C853 P39577 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 671988 T3232 C853 P39577 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 671994 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10051 (1438202839646076929)]} 0 1
[junit4:junit4] 2> 671994 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[51 (1438202839646076928)]} 0 1
[junit4:junit4] 2> 671998 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10051 (1438202839649222656)]} 0 1
[junit4:junit4] 2> 671998 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1438202839649222657)]} 0 0
[junit4:junit4] 2> 671999 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10051 (1438202839649222656)]} 0 3
[junit4:junit4] 2> 671999 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[51 (1438202839649222657)]} 0 3
[junit4:junit4] 2> 672005 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10046 (-1438202839658659840)]} 0 0
[junit4:junit4] 2> 672005 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[46 (-1438202839658659841)]} 0 0
[junit4:junit4] 2> 672009 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10046 (-1438202839660756992)]} 0 1
[junit4:junit4] 2> 672009 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[46 (-1438202839660756993)]} 0 1
[junit4:junit4] 2> 672009 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10046 (-1438202839660756992)]} 0 2
[junit4:junit4] 2> 672010 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[46 (-1438202839660756993)]} 0 3
[junit4:junit4] 2> 672012 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10052 (1438202839664951296)]} 0 1
[junit4:junit4] 2> 672012 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[52 (1438202839664951297)]} 0 1
[junit4:junit4] 2> 672015 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10052 (1438202839667048448)]} 0 0
[junit4:junit4] 2> 672016 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1438202839668097024)]} 0 1
[junit4:junit4] 2> 672016 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10052 (1438202839667048448)]} 0 3
[junit4:junit4] 2> 672016 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[52 (1438202839668097024)]} 0 2
[junit4:junit4] 2> 672098 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10053 (1438202839755128832)]} 0 1
[junit4:junit4] 2> 672098 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[53 (1438202839755128833)]} 0 1
[junit4:junit4] 2> 672103 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10053 (1438202839758274560)]} 0 0
[junit4:junit4] 2> 672104 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1438202839759323136)]} 0 1
[junit4:junit4] 2> 672104 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10053 (1438202839758274560)]} 0 4
[junit4:junit4] 2> 672105 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[53 (1438202839759323136)]} 0 4
[junit4:junit4] 2> 672178 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10047 (-1438202839839014912)]} 0 1
[junit4:junit4] 2> 672178 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[47 (-1438202839840063488)]} 0 0
[junit4:junit4] 2> 672182 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10047 (-1438202839842160640)]} 0 0
[junit4:junit4] 2> 672183 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[47 (-1438202839843209216)]} 0 1
[junit4:junit4] 2> 672183 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10047 (-1438202839842160640)]} 0 3
[junit4:junit4] 2> 672184 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[47 (-1438202839843209216)]} 0 3
[junit4:junit4] 2> 672187 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10054 (1438202839848452096)]} 0 1
[junit4:junit4] 2> 672188 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[54 (1438202839849500672)]} 0 1
[junit4:junit4] 2> 672193 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10054 (1438202839852646400)]} 0 1
[junit4:junit4] 2> 672194 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1438202839852646401)]} 0 1
[junit4:junit4] 2> 672194 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10054 (1438202839852646400)]} 0 5
[junit4:junit4] 2> 672195 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[54 (1438202839852646401)]} 0 5
[junit4:junit4] 2> 672211 T3194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 672212 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:39577_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39577/tp/au"}
[junit4:junit4] 2> 672216 T3194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39577_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39577/tp/au"}
[junit4:junit4] 2> 672221 T3193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672222 T3229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672222 T3213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672222 T3200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672263 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10048 (-1438202839929192448)]} 0 0
[junit4:junit4] 2> 672264 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[48 (-1438202839930241024)]} 0 1
[junit4:junit4] 2> C853_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:39577_tp%2Fau, base_url=http://127.0.0.1:39577/tp/au}
[junit4:junit4] 2> 672268 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10048 (-1438202839932338176)]} 0 1
[junit4:junit4] 2> 672268 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[48 (-1438202839932338177)]} 0 0
[junit4:junit4] 2> 672269 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10048 (-1438202839932338176)]} 0 3
[junit4:junit4] 2> 672269 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[48 (-1438202839932338177)]} 0 3
[junit4:junit4] 2> 672272 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10055 (1438202839938629632)]} 0 1
[junit4:junit4] 2> 672273 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[55 (1438202839938629633)]} 0 1
[junit4:junit4] 2> 672278 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10055 (1438202839941775360)]} 0 1
[junit4:junit4] 2> 672279 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1438202839941775361)]} 0 1
[junit4:junit4] 2> 672279 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10055 (1438202839941775360)]} 0 4
[junit4:junit4] 2> 672279 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[55 (1438202839941775361)]} 0 4
[junit4:junit4] 2> 672313 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10056 (1438202839981621248)]} 0 0
[junit4:junit4] 2> 672314 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[56 (1438202839982669824)]} 0 1
[junit4:junit4] 2> 672319 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10056 (1438202839984766976)]} 0 1
[junit4:junit4] 2> 672319 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1438202839985815552)]} 0 0
[junit4:junit4] 2> 672320 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10056 (1438202839984766976)]} 0 4
[junit4:junit4] 2> 672320 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[56 (1438202839985815552)]} 0 4
[junit4:junit4] 2> 672373 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10057 (1438202840043487232)]} 0 1
[junit4:junit4] 2> 672373 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[57 (1438202840043487233)]} 0 1
[junit4:junit4] 2> 672378 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10057 (1438202840046632960)]} 0 1
[junit4:junit4] 2> 672379 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1438202840047681536)]} 0 1
[junit4:junit4] 2> 672379 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10057 (1438202840046632960)]} 0 4
[junit4:junit4] 2> 672380 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[57 (1438202840047681536)]} 0 4
[junit4:junit4] 2> 672467 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10058 (1438202840142053376)]} 0 1
[junit4:junit4] 2> 672467 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[58 (1438202840142053377)]} 0 1
[junit4:junit4] 2> 672471 T3224 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10058 (1438202840144150528)]} 0 1
[junit4:junit4] 2> 672471 T3222 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1438202840145199104)]} 0 1
[junit4:junit4] 2> 672471 T3208 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10058 (1438202840144150528)]} 0 3
[junit4:junit4] 2> 672471 T3207 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[58 (1438202840145199104)]} 0 2
[junit4:junit4] 2> 672507 T3187 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10059 (1438202840185044992)]} 0 1
[junit4:junit4] 2> 672507 T3189 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[59 (1438202840185044993)]} 0 1
[junit4:junit4] 2> 672511 T3225 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10059 (1438202840187142144)]} 0 0
[junit4:junit4] 2> 672512 T3223 C853 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={distrib.from=http://127.0.0.1:48395/tp/au/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1438202840187142145)]} 0 1
[junit4:junit4] 2> 672512 T3206 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[10059 (1438202840187142144)]} 0 3
[junit4:junit4] 2> 672512 T3209 C854 P48395 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {add=[59 (1438202840187142145)]} 0 3
[junit4:junit4] 2> 672532 T3188 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] webapp=/tp/au path=/update params={wt=javabin&version=2} {delete=[10050 (-1438202840211259392)]} 0 0
[junit4:junit4] 2> 672532 T3186 C852 P60338 oasup.LogUpdateProcessor.finish [collection1] w
[...truncated too long message...]
unit4] 2> 794865 T3253 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 794866 T3253 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/tp/au/collection1/ has no replicas
[junit4:junit4] 2> 794866 T3253 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39577/tp/au/collection1/
[junit4:junit4] 2> 794866 T3263 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48395_tp%2Fau",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48395/tp/au"}
[junit4:junit4] 2> 794866 T3253 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 794872 T3253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 794877 T3171 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/tp/au,null}
[junit4:junit4] 2> 794928 T3171 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 39577
[junit4:junit4] 2> 794929 T3171 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=32897152
[junit4:junit4] 2> 795931 T3171 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 795932 T3171 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 795933 T3171 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@32eaf5
[junit4:junit4] 2> 795945 T3171 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=52,cumulative_deletesById=26,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 795946 T3171 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 795947 T3171 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 795947 T3171 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 795949 T3171 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 795949 T3171 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 795950 T3171 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2;done=false>>]
[junit4:junit4] 2> 795950 T3171 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2
[junit4:junit4] 2> 795951 T3171 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index;done=false>>]
[junit4:junit4] 2> 795951 T3171 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371577097377/jetty2/index
[junit4:junit4] 2> 795952 T3263 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89887676653961226-127.0.0.1:39577_tp%2Fau-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 795973 T3171 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/tp/au,null}
[junit4:junit4] 2> 796030 T3171 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 796032 T3171 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35894 35894
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=B9548118778436 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ga_IE -Dtests.timezone=Poland -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 137s J0 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<620> but was:<52>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([B9548118778436:815FDA996F28E40A]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 796062 T3171 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 136652 T3170 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> 797253 T3253 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 797253 T3253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 797254 T3253 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=NestedPulsing), text=PostingsFormat(name=SimpleText), _version_=Pulsing41(freqCutoff=20 minBlockSize=42 maxBlockSize=111), rnd_b=PostingsFormat(name=Lucene41WithOrds), intDefault=Pulsing41(freqCutoff=20 minBlockSize=42 maxBlockSize=111), timestamp=PostingsFormat(name=NestedPulsing), id=PostingsFormat(name=Lucene41WithOrds), a_t=PostingsFormat(name=NestedPulsing), range_facet_sl=PostingsFormat(name=Lucene41WithOrds), range_facet_si=PostingsFormat(name=NestedPulsing), other_tl1=Pulsing41(freqCutoff=20 minBlockSize=42 maxBlockSize=111), multiDefault=PostingsFormat(name=NestedPulsing), a_si=Pulsing41(freqCutoff=20 minBlockSize=42 maxBlockSize=111)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ga_IE, timezone=Poland
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_21 (32-bit)/cpus=8,threads=1,free=152126640,total=227278848
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSearchPerf, BasicDistributedZkTest, TestGroupingSearch, TestLRUCache, CacheHeaderTest, TestIBSimilarityFactory, TestSchemaResource, TestSchemaSimilarityResource, TestBadConfig, TestFastWriter, TestReplicationHandler, LoggingHandlerTest, TestExtendedDismaxParser, TermsComponentTest, TestLFUCache, PluginInfoTest, TestManagedSchema, BasicFunctionalityTest, SoftAutoCommitTest, TestPluginEnable, TestSolrXmlPersistence, FileBasedSpellCheckerTest, ResourceLoaderTest, SuggesterFSTTest, TestClassNameShortening, SearchHandlerTest, TestDocSet, TestPHPSerializedResponseWriter, SimpleFacetsTest, TestSolrCoreProperties, TestStressVersions, TestFastLRUCache, AlternateDirectoryTest, DistributedTermsComponentTest, QueryElevationComponentTest, CircularListTest, TestSolrDeletionPolicy1, FieldMutatingUpdateProcessorTest, TestPseudoReturnFields, AutoCommitTest, TestFuzzyAnalyzedSuggestions, TestCloudManagedSchema, DocValuesTest, TestReloadAndDeleteDocs, TestSolrDeletionPolicy2, TestRealTimeGet, SuggesterTest, DirectUpdateHandlerOptimizeTest, SliceStateTest, TestCharFilters, BinaryUpdateRequestHandlerTest, SolrInfoMBeanTest, ScriptEngineTest, TermVectorComponentTest, LukeRequestHandlerTest, MBeansHandlerTest, SolrIndexSplitterTest, SynonymTokenizerTest, TestSurroundQueryParser, JsonLoaderTest, TestXIncludeConfig, BadComponentTest, TestQueryUtils, TestUniqueKeyFieldResource, TestPhraseSuggestions, ZkControllerTest, DistributedQueryElevationComponentTest, UpdateRequestProcessorFactoryTest, RegexBoostProcessorTest, TestJoin, WordBreakSolrSpellCheckerTest, TestCoreContainer, UnloadDistributedZkTest, SyncSliceTest, OverseerTest, ShardRoutingTest, RecoveryZkTest]
[junit4:junit4] Completed on J0 in 137.95s, 1 test, 1 failure <<< FAILURES!
[...truncated 475 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1250 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 38 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_21 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6199 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6199/
Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:33880/r/collection1lastClient and got 266 from http://127.0.0.1:45699/r/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:33880/r/collection1lastClient and got 266 from http://127.0.0.1:45699/r/collection1
at __randomizedtesting.SeedInfo.seed([14FA98A28E8105C6:951C16BAF9DE65FA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9577 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 443369 T1801 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /r/
[junit4:junit4] 2> 443374 T1801 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1371596432910
[junit4:junit4] 2> 443375 T1801 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 443376 T1802 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 443476 T1801 oasc.ZkTestServer.run start zk server on port:54259
[junit4:junit4] 2> 443477 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443530 T1808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1335018 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443531 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443531 T1801 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 443538 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443541 T1810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f4ab0e name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443541 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443542 T1801 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 443547 T1801 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 443551 T1801 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 443554 T1801 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 443557 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 443558 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 443562 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 443563 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 443570 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 443571 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 443574 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 443574 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 443577 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 443577 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 443580 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 443580 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 443582 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 443583 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 443585 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 443585 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 443588 T1801 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 443589 T1801 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 443593 T1803 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f5984a6100001, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 443664 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 443672 T1801 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40647
[junit4:junit4] 2> 443673 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 443674 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 443674 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127
[junit4:junit4] 2> 443674 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/solr.xml
[junit4:junit4] 2> 443675 T1801 oasc.CoreContainer.<init> New CoreContainer 27867822
[junit4:junit4] 2> 443675 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/'
[junit4:junit4] 2> 443675 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/'
[junit4:junit4] 2> 443723 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 443724 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 443724 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 443724 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 443725 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 443725 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 443725 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 443726 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 443726 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 443726 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 443729 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 443730 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
[junit4:junit4] 2> 443730 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 443731 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443733 T1821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b38c68 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443733 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443735 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 443738 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443741 T1823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11f7408 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443741 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443743 T1801 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 443745 T1801 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 443747 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 443749 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40647_r
[junit4:junit4] 2> 443750 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40647_r
[junit4:junit4] 2> 443752 T1801 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 443758 T1801 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 443760 T1801 oasc.Overseer.start Overseer (id=89888943827320835-127.0.0.1:40647_r-n_0000000000) starting
[junit4:junit4] 2> 443764 T1801 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 443766 T1825 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 443767 T1801 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 443769 T1801 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 443770 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 443772 T1824 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 443778 T1826 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 443778 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 445275 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 445275 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:40647_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40647/r"}
[junit4:junit4] 2> 445276 T1824 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 445276 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 445296 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 445779 T1826 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/collection1
[junit4:junit4] 2> 445780 T1826 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 445781 T1826 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 445781 T1826 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 445782 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/collection1/'
[junit4:junit4] 2> 445782 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/collection1/lib/README' to classloader
[junit4:junit4] 2> 445783 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 445813 T1826 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 445850 T1826 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 445852 T1826 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 445856 T1826 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 446251 T1826 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 446257 T1826 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 446259 T1826 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 446270 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 446273 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 446276 T1826 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 446277 T1826 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 446277 T1826 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 446277 T1826 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 446278 T1826 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 446278 T1826 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 446278 T1826 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 446286 T1826 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371596433127/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/
[junit4:junit4] 2> 446286 T1826 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48a8dd
[junit4:junit4] 2> 446286 T1826 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 446287 T1826 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data
[junit4:junit4] 2> 446287 T1826 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index/
[junit4:junit4] 2> 446287 T1826 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 446287 T1826 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index
[junit4:junit4] 2> 446293 T1826 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 446294 T1826 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 446296 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 446297 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 446297 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 446298 T1826 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 446298 T1826 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 446299 T1826 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 446299 T1826 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 446300 T1826 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 446300 T1826 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 446302 T1826 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 446307 T1826 oass.SolrIndexSearcher.<init> Opening Searcher@1fea415 main
[junit4:junit4] 2> 446308 T1826 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 446308 T1826 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 446312 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1fea415 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 446314 T1826 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 446314 T1826 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40647/r collection:control_collection shard:shard1
[junit4:junit4] 2> 446315 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 446322 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 446324 T1826 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 446324 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 446325 T1826 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40647/r/collection1/
[junit4:junit4] 2> 446325 T1826 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 446325 T1826 oasc.SyncStrategy.syncToMe http://127.0.0.1:40647/r/collection1/ has no replicas
[junit4:junit4] 2> 446325 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40647/r/collection1/
[junit4:junit4] 2> 446326 T1826 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 446799 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446806 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 446831 T1826 oasc.ZkController.register We are http://127.0.0.1:40647/r/collection1/ and leader is http://127.0.0.1:40647/r/collection1/
[junit4:junit4] 2> 446831 T1826 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40647/r
[junit4:junit4] 2> 446832 T1826 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 446832 T1826 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 446832 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 446833 T1826 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446834 T1801 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 446835 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 446835 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 446840 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 446841 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446843 T1830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69e0d2 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446843 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446844 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 446846 T1801 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 446915 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 446917 T1801 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33880
[junit4:junit4] 2> 446919 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 446919 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 446919 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381
[junit4:junit4] 2> 446920 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/solr.xml
[junit4:junit4] 2> 446920 T1801 oasc.CoreContainer.<init> New CoreContainer 23752950
[junit4:junit4] 2> 446920 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/'
[junit4:junit4] 2> 446921 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/'
[junit4:junit4] 2> 446964 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 446965 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 446965 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 446965 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 446966 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 446966 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 446966 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 446967 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 446967 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 446967 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 446970 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 446971 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
[junit4:junit4] 2> 446971 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 446972 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446973 T1841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f0f72c name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446974 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446975 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 446978 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446979 T1843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1780e82 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446980 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446983 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 447985 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33880_r
[junit4:junit4] 2> 447986 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33880_r
[junit4:junit4] 2> 447989 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 447989 T1830 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 447989 T1843 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 447990 T1823 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 447994 T1844 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 447995 T1844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 448310 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 448311 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:40647_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40647/r"}
[junit4:junit4] 2> 448314 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:33880_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33880/r"}
[junit4:junit4] 2> 448314 T1824 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 448314 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 448319 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 448319 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 448320 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 448996 T1844 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/collection1
[junit4:junit4] 2> 448996 T1844 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 448997 T1844 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 448997 T1844 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 448998 T1844 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/collection1/'
[junit4:junit4] 2> 448998 T1844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/collection1/lib/README' to classloader
[junit4:junit4] 2> 448999 T1844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 449028 T1844 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 449069 T1844 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 449071 T1844 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 449074 T1844 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 449463 T1844 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 449469 T1844 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 449471 T1844 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 449488 T1844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 449491 T1844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 449494 T1844 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 449495 T1844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 449495 T1844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 449495 T1844 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 449497 T1844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 449497 T1844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 449497 T1844 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 449497 T1844 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371596436381/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/
[junit4:junit4] 2> 449497 T1844 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48a8dd
[junit4:junit4] 2> 449498 T1844 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 449498 T1844 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1
[junit4:junit4] 2> 449498 T1844 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index/
[junit4:junit4] 2> 449499 T1844 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 449499 T1844 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index
[junit4:junit4] 2> 449504 T1844 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 449504 T1844 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 449507 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 449507 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 449508 T1844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 449509 T1844 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 449519 T1844 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 449520 T1844 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 449520 T1844 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 449520 T1844 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 449521 T1844 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 449522 T1844 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 449525 T1844 oass.SolrIndexSearcher.<init> Opening Searcher@107181 main
[junit4:junit4] 2> 449526 T1844 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 449526 T1844 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 449533 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@107181 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 449534 T1844 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 449534 T1844 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33880/r collection:collection1 shard:shard1
[junit4:junit4] 2> 449535 T1844 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 449541 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 449544 T1844 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 449544 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 449544 T1844 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 449545 T1844 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 449545 T1844 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/ has no replicas
[junit4:junit4] 2> 449545 T1844 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 449545 T1844 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 449825 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449830 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 449830 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 449830 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 449852 T1844 oasc.ZkController.register We are http://127.0.0.1:33880/r/collection1/ and leader is http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 449852 T1844 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33880/r
[junit4:junit4] 2> 449853 T1844 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 449853 T1844 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 449853 T1844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 449857 T1844 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449858 T1801 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 449859 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 449859 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 449958 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 449962 T1801 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50641
[junit4:junit4] 2> 449963 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 449963 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 449964 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399
[junit4:junit4] 2> 449964 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/solr.xml
[junit4:junit4] 2> 449964 T1801 oasc.CoreContainer.<init> New CoreContainer 26462235
[junit4:junit4] 2> 449965 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/'
[junit4:junit4] 2> 449965 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/'
[junit4:junit4] 2> 450021 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 450022 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 450022 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 450022 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 450022 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 450023 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 450023 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 450023 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 450024 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 450024 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 450027 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 450028 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
[junit4:junit4] 2> 450028 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 450029 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 450032 T1857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ad9c3 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 450032 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 450034 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 450041 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 450042 T1859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff0a0a name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 450043 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 450046 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 451049 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50641_r
[junit4:junit4] 2> 451050 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50641_r
[junit4:junit4] 2> 451053 T1830 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 451053 T1843 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 451053 T1859 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 451054 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451053 T1823 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 451054 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451053 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451059 T1860 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 451059 T1860 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 451335 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 451336 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:33880_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33880/r"}
[junit4:junit4] 2> 451339 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:50641_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50641/r"}
[junit4:junit4] 2> 451339 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 451339 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 451342 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451342 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451342 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 451342 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452061 T1860 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/collection1
[junit4:junit4] 2> 452061 T1860 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 452061 T1860 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 452062 T1860 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 452062 T1860 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/collection1/'
[junit4:junit4] 2> 452063 T1860 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/collection1/lib/README' to classloader
[junit4:junit4] 2> 452063 T1860 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 452092 T1860 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 452124 T1860 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 452125 T1860 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 452129 T1860 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 452506 T1860 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 452511 T1860 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 452514 T1860 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 452527 T1860 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 452529 T1860 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 452532 T1860 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 452533 T1860 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 452534 T1860 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 452534 T1860 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 452535 T1860 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 452535 T1860 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 452535 T1860 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 452535 T1860 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371596439399/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/
[junit4:junit4] 2> 452535 T1860 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48a8dd
[junit4:junit4] 2> 452536 T1860 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 452536 T1860 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2
[junit4:junit4] 2> 452537 T1860 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index/
[junit4:junit4] 2> 452537 T1860 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 452537 T1860 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index
[junit4:junit4] 2> 452541 T1860 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 452541 T1860 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 452543 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 452543 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 452544 T1860 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 452544 T1860 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 452545 T1860 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 452545 T1860 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 452545 T1860 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 452546 T1860 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 452546 T1860 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 452547 T1860 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 452550 T1860 oass.SolrIndexSearcher.<init> Opening Searcher@1a08211 main
[junit4:junit4] 2> 452551 T1860 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 452551 T1860 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 452556 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a08211 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 452558 T1860 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 452558 T1860 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50641/r collection:collection1 shard:shard1
[junit4:junit4] 2> 452561 T1860 oasc.ZkController.register We are http://127.0.0.1:50641/r/collection1/ and leader is http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 452562 T1860 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50641/r
[junit4:junit4] 2> 452562 T1860 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 452562 T1860 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C950 name=collection1 org.apache.solr.core.SolrCore@e8c4fb url=http://127.0.0.1:50641/r/collection1 node=127.0.0.1:50641_r C950_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, state=down, node_name=127.0.0.1:50641_r, base_url=http://127.0.0.1:50641/r}
[junit4:junit4] 2> 452562 T1862 C950 P50641 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 452563 T1860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452563 T1862 C950 P50641 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 452563 T1862 C950 P50641 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 452563 T1862 C950 P50641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 452564 T1801 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 452564 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 452565 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 452567 T1862 C950 P50641 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 452570 T1837 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 452637 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 452639 T1801 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45699
[junit4:junit4] 2> 452640 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 452640 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 452641 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103
[junit4:junit4] 2> 452641 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/solr.xml
[junit4:junit4] 2> 452641 T1801 oasc.CoreContainer.<init> New CoreContainer 11033838
[junit4:junit4] 2> 452642 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/'
[junit4:junit4] 2> 452642 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/'
[junit4:junit4] 2> 452686 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 452686 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 452687 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 452687 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 452687 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 452688 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 452688 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 452688 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 452689 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 452689 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 452703 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 452704 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
[junit4:junit4] 2> 452704 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 452705 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 452706 T1874 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a68c9 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452707 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 452709 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 452712 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 452713 T1876 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65010f name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452714 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 452718 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 452848 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452849 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "node_name":"127.0.0.1:50641_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50641/r"}
[junit4:junit4] 2> 452852 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452852 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452852 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452852 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 452852 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 453571 T1837 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 453571 T1837 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&onlyIfLeader=true&core=collection1&checkLive=true&nodeName=127.0.0.1:50641_r&wt=javabin&state=recovering&version=2&coreNodeName=3} status=0 QTime=1001
[junit4:junit4] 2> 453721 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45699_r
[junit4:junit4] 2> 453722 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45699_r
[junit4:junit4] 2> 453724 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 453724 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 453724 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 453724 T1843 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453724 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 453725 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 453725 T1859 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453726 T1823 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453726 T1876 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453726 T1830 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 453729 T1877 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 453730 T1877 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 454356 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 454357 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:45699_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45699/r"}
[junit4:junit4] 2> 454357 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 454357 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 454361 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 454361 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 454361 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 454361 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 454361 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 454734 T1877 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/collection1
[junit4:junit4] 2> 454734 T1877 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 454735 T1877 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 454735 T1877 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 454736 T1877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/collection1/'
[junit4:junit4] 2> 454737 T1877 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/collection1/lib/README' to classloader
[junit4:junit4] 2> 454737 T1877 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 454782 T1877 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 454832 T1877 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 454834 T1877 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 454839 T1877 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 455247 T1877 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 455252 T1877 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 455255 T1877 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 455265 T1877 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 455268 T1877 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 455271 T1877 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 455271 T1877 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 455272 T1877 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 455272 T1877 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 455273 T1877 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 455273 T1877 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 455274 T1877 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 455274 T1877 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371596442103/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/
[junit4:junit4] 2> 455274 T1877 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48a8dd
[junit4:junit4] 2> 455274 T1877 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 455275 T1877 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3
[junit4:junit4] 2> 455275 T1877 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index/
[junit4:junit4] 2> 455275 T1877 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 455276 T1877 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index
[junit4:junit4] 2> 455280 T1877 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 455281 T1877 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 455282 T1877 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 455283 T1877 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 455283 T1877 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 455284 T1877 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 455284 T1877 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 455284 T1877 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 455285 T1877 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 455285 T1877 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 455285 T1877 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 455287 T1877 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 455289 T1877 oass.SolrIndexSearcher.<init> Opening Searcher@11535ab main
[junit4:junit4] 2> 455290 T1877 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 455290 T1877 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 455294 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11535ab main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 455295 T1877 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 455295 T1877 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45699/r collection:collection1 shard:shard1
[junit4:junit4] 2> 455298 T1877 oasc.ZkController.register We are http://127.0.0.1:45699/r/collection1/ and leader is http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 455298 T1877 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45699/r
[junit4:junit4] 2> 455298 T1877 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 455299 T1877 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C951 name=collection1 org.apache.solr.core.SolrCore@277f02 url=http://127.0.0.1:45699/r/collection1 node=127.0.0.1:45699_r C951_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, state=down, node_name=127.0.0.1:45699_r, base_url=http://127.0.0.1:45699/r}
[junit4:junit4] 2> 455299 T1879 C951 P45699 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 455300 T1879 C951 P45699 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 455300 T1877 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 455300 T1879 C951 P45699 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 455300 T1879 C951 P45699 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 455301 T1801 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 455301 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 455301 T1879 C951 P45699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 455301 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 455307 T1836 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 455369 T1801 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 455371 T1801 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60000
[junit4:junit4] 2> 455371 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 455372 T1801 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 455372 T1801 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841
[junit4:junit4] 2> 455372 T1801 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/solr.xml
[junit4:junit4] 2> 455373 T1801 oasc.CoreContainer.<init> New CoreContainer 16118769
[junit4:junit4] 2> 455373 T1801 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/'
[junit4:junit4] 2> 455373 T1801 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/'
[junit4:junit4] 2> 455418 T1801 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 455419 T1801 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 455419 T1801 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 455419 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 455420 T1801 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 455420 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 455420 T1801 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 455420 T1801 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 455421 T1801 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 455421 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 455424 T1801 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 455425 T1801 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54259/solr
[junit4:junit4] 2> 455425 T1801 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 455426 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 455427 T1891 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1221191 name:ZooKeeperConnection Watcher:127.0.0.1:54259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 455428 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 455429 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 455432 T1801 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 455433 T1893 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f712a0 name:ZooKeeperConnection Watcher:127.0.0.1:54259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 455433 T1801 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 455436 T1801 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C952 name=collection1 org.apache.solr.core.SolrCore@e8c4fb url=http://127.0.0.1:50641/r/collection1 node=127.0.0.1:50641_r C952_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=recovering, node_name=127.0.0.1:50641_r, base_url=http://127.0.0.1:50641/r}
[junit4:junit4] 2> 455572 T1862 C952 P50641 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33880/r/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 455572 T1862 C952 P50641 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50641/r START replicas=[http://127.0.0.1:33880/r/collection1/] nUpdates=100
[junit4:junit4] 2> 455573 T1862 C952 P50641 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 455573 T1862 C952 P50641 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 455574 T1862 C952 P50641 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 455574 T1862 C952 P50641 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 455574 T1862 C952 P50641 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 455574 T1862 C952 P50641 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33880/r/collection1/. core=collection1
[junit4:junit4] 2> 455575 T1862 C952 P50641 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C953 name=collection1 org.apache.solr.core.SolrCore@1f0016a url=http://127.0.0.1:33880/r/collection1 node=127.0.0.1:33880_r C953_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=active, node_name=127.0.0.1:33880_r, base_url=http://127.0.0.1:33880/r, leader=true}
[junit4:junit4] 2> 455578 T1837 C953 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 455580 T1839 C953 P33880 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 455581 T1839 C953 P33880 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 455582 T1839 C953 P33880 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 455589 T1839 C953 P33880 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 455589 T1839 C953 P33880 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 455590 T1839 C953 P33880 oass.SolrIndexSearcher.<init> Opening Searcher@7bbb17 realtime
[junit4:junit4] 2> 455590 T1839 C953 P33880 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 455590 T1839 C953 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&openSearcher=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 10
[junit4:junit4] 2> 455591 T1862 C952 P50641 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 455591 T1862 C952 P50641 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 455595 T1837 C953 P33880 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 455595 T1837 C953 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
[junit4:junit4] 2> 455596 T1862 C952 P50641 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 455596 T1862 C952 P50641 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 455596 T1862 C952 P50641 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 455598 T1839 C953 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=filelist&qt=/replication&generation=2&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 455599 T1862 C952 P50641 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 455600 T1862 C952 P50641 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index.20130618200045134
[junit4:junit4] 2> 455600 T1862 C952 P50641 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index.20130618200045134 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1dc0b8; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 455603 T1838 C953 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=filecontent&checksum=true&qt=/replication&generation=2&wt=filestream&file=segments_2} status=0 QTime=0
[junit4:junit4] 2> 455604 T1862 C952 P50641 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 455609 T1862 C952 P50641 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 455609 T1862 C952 P50641 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 455611 T1862 C952 P50641 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 455611 T1862 C952 P50641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 455612 T1862 C952 P50641 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 455612 T1862 C952 P50641 oass.SolrIndexSearcher.<init> Opening Searcher@ebe0cf main
[junit4:junit4] 2> 455613 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ebe0cf main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 455613 T1862 C952 P50641 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index.20130618200045134 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index.20130618200045134;done=true>>]
[junit4:junit4] 2> 455613 T1862 C952 P50641 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index.20130618200045134
[junit4:junit4] 2> 455614 T1862 C952 P50641 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index.20130618200045134
[junit4:junit4] 2> 455614 T1862 C952 P50641 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 455614 T1862 C952 P50641 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 455614 T1862 C952 P50641 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 455614 T1862 C952 P50641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 455616 T1862 C952 P50641 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 455866 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 455867 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "node_name":"127.0.0.1:45699_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45699/r"}
[junit4:junit4] 2> 455870 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:50641_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50641/r"}
[junit4:junit4] 2> 455889 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455889 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455889 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455889 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455889 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 455889 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 456307 T1836 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 456308 T1836 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&onlyIfLeader=true&core=collection1&checkLive=true&nodeName=127.0.0.1:45699_r&wt=javabin&state=recovering&version=2&coreNodeName=4} status=0 QTime=1001
[junit4:junit4] 2> 456439 T1801 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60000_r
[junit4:junit4] 2> 456441 T1801 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60000_r
[junit4:junit4] 2> 456444 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 456444 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 456444 T1830 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456444 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 456445 T1859 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456445 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456444 T1893 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456445 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456446 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456447 T1876 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456447 T1843 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456447 T1823 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 456451 T1896 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 456451 T1896 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 457396 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 457397 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "node_name":"127.0.0.1:60000_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60000/r"}
[junit4:junit4] 2> 457397 T1824 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 457397 T1824 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 457402 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457402 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457402 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457402 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457402 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457402 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 457452 T1896 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/collection1
[junit4:junit4] 2> 457453 T1896 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 457453 T1896 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 457454 T1896 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 457455 T1896 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/collection1/'
[junit4:junit4] 2> 457455 T1896 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/collection1/lib/README' to classloader
[junit4:junit4] 2> 457456 T1896 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 457482 T1896 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 457528 T1896 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 457530 T1896 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 457533 T1896 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 457865 T1896 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 457870 T1896 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 457873 T1896 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 457883 T1896 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 457886 T1896 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 457888 T1896 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 457889 T1896 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 457890 T1896 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 457890 T1896 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 457891 T1896 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 457891 T1896 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 457891 T1896 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 457891 T1896 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371596444841/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/
[junit4:junit4] 2> 457892 T1896 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48a8dd
[junit4:junit4] 2> 457892 T1896 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 457892 T1896 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4
[junit4:junit4] 2> 457893 T1896 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index/
[junit4:junit4] 2> 457893 T1896 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 457893 T1896 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index
[junit4:junit4] 2> 457896 T1896 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 457896 T1896 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 457898 T1896 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 457898 T1896 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 457899 T1896 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 457899 T1896 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 457900 T1896 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 457900 T1896 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 457900 T1896 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 457901 T1896 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 457901 T1896 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 457903 T1896 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 457905 T1896 oass.SolrIndexSearcher.<init> Opening Searcher@1f1e214 main
[junit4:junit4] 2> 457906 T1896 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 457906 T1896 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 457910 T1897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f1e214 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 457912 T1896 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 457912 T1896 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60000/r collection:collection1 shard:shard1
[junit4:junit4] 2> 457914 T1896 oasc.ZkController.register We are http://127.0.0.1:60000/r/collection1/ and leader is http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 457914 T1896 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60000/r
[junit4:junit4] 2> 457914 T1896 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 457915 T1896 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C954 name=collection1 org.apache.solr.core.SolrCore@ef02d8 url=http://127.0.0.1:60000/r/collection1 node=127.0.0.1:60000_r C954_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, state=down, node_name=127.0.0.1:60000_r, base_url=http://127.0.0.1:60000/r}
[junit4:junit4] 2> 457915 T1898 C954 P60000 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 457915 T1896 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 457915 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 457916 T1898 C954 P60000 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 457916 T1898 C954 P60000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 457916 T1801 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 457917 T1801 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 457917 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 457917 T1898 C954 P60000 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 457920 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 457921 T1839 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 457921 T1801 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 457921 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 457922 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C955 name=collection1 org.apache.solr.core.SolrCore@277f02 url=http://127.0.0.1:45699/r/collection1 node=127.0.0.1:45699_r C955_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=recovering, node_name=127.0.0.1:45699_r, base_url=http://127.0.0.1:45699/r}
[junit4:junit4] 2> 458308 T1879 C955 P45699 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33880/r/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 458309 T1879 C955 P45699 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45699/r START replicas=[http://127.0.0.1:33880/r/collection1/] nUpdates=100
[junit4:junit4] 2> 458309 T1879 C955 P45699 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 458309 T1879 C955 P45699 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 458309 T1879 C955 P45699 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 458309 T1879 C955 P45699 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 458310 T1879 C955 P45699 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 458310 T1879 C955 P45699 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33880/r/collection1/. core=collection1
[junit4:junit4] 2> 458310 T1879 C955 P45699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C956 name=collection1 org.apache.solr.core.SolrCore@1f0016a url=http://127.0.0.1:33880/r/collection1 node=127.0.0.1:33880_r C956_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=active, node_name=127.0.0.1:33880_r, base_url=http://127.0.0.1:33880/r, leader=true}
[junit4:junit4] 2> 458311 T1838 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 458313 T1837 C956 P33880 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 458319 T1837 C956 P33880 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 458319 T1837 C956 P33880 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 458319 T1837 C956 P33880 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 458320 T1837 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&openSearcher=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 7
[junit4:junit4] 2> 458320 T1879 C955 P45699 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 458320 T1879 C955 P45699 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 458322 T1838 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 458323 T1879 C955 P45699 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 458323 T1879 C955 P45699 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 458323 T1879 C955 P45699 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 458325 T1837 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=filelist&qt=/replication&generation=3&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 458325 T1879 C955 P45699 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 458326 T1879 C955 P45699 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index.20130618200047860
[junit4:junit4] 2> 458326 T1879 C955 P45699 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index.20130618200047860 lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6cb07; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 458328 T1836 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=filecontent&checksum=true&qt=/replication&generation=3&wt=filestream&file=segments_3} status=0 QTime=0
[junit4:junit4] 2> 458329 T1879 C955 P45699 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 458332 T1879 C955 P45699 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 458332 T1879 C955 P45699 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 458333 T1879 C955 P45699 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 458334 T1879 C955 P45699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 458334 T1879 C955 P45699 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 458334 T1879 C955 P45699 oass.SolrIndexSearcher.<init> Opening Searcher@1199875 main
[junit4:junit4] 2> 458335 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1199875 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 458336 T1879 C955 P45699 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index.20130618200047860 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index.20130618200047860;done=true>>]
[junit4:junit4] 2> 458336 T1879 C955 P45699 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index.20130618200047860
[junit4:junit4] 2> 458336 T1879 C955 P45699 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index.20130618200047860
[junit4:junit4] 2> 458336 T1879 C955 P45699 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 458337 T1879 C955 P45699 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 458337 T1879 C955 P45699 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 458337 T1879 C955 P45699 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 458338 T1879 C955 P45699 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 458906 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 458907 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "node_name":"127.0.0.1:60000_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60000/r"}
[junit4:junit4] 2> 458910 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:45699_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45699/r"}
[junit4:junit4] 2> 458912 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458913 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458913 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458913 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458913 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458913 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 458921 T1839 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 458922 T1839 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&onlyIfLeader=true&core=collection1&checkLive=true&nodeName=127.0.0.1:60000_r&wt=javabin&state=recovering&version=2&coreNodeName=5} status=0 QTime=1001
[junit4:junit4] 2> 458923 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 459925 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C954_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=recovering, node_name=127.0.0.1:60000_r, base_url=http://127.0.0.1:60000/r}
[junit4:junit4] 2> 460922 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33880/r/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 460923 T1898 C954 P60000 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60000/r START replicas=[http://127.0.0.1:33880/r/collection1/] nUpdates=100
[junit4:junit4] 2> 460923 T1898 C954 P60000 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 460923 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 460923 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 460923 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 460924 T1898 C954 P60000 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 460924 T1898 C954 P60000 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33880/r/collection1/. core=collection1
[junit4:junit4] 2> 460924 T1898 C954 P60000 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 460925 T1837 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 460926 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 460928 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 460939 T1836 C956 P33880 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 460939 T1836 C956 P33880 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 460940 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 460940 T1836 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&openSearcher=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 13
[junit4:junit4] 2> 460940 T1898 C954 P60000 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 460941 T1898 C954 P60000 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 460942 T1839 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 460943 T1898 C954 P60000 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 460943 T1898 C954 P60000 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 460943 T1898 C954 P60000 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 460945 T1837 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=filelist&qt=/replication&generation=4&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 460945 T1898 C954 P60000 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 460946 T1898 C954 P60000 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index.20130618200050481
[junit4:junit4] 2> 460947 T1898 C954 P60000 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index.20130618200050481 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ea6553; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 460949 T1836 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/replication params={command=filecontent&checksum=true&qt=/replication&generation=4&wt=filestream&file=segments_4} status=0 QTime=0
[junit4:junit4] 2> 460950 T1898 C954 P60000 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 460951 T1898 C954 P60000 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 460951 T1898 C954 P60000 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 460953 T1898 C954 P60000 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 460954 T1898 C954 P60000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 460954 T1898 C954 P60000 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 460954 T1898 C954 P60000 oass.SolrIndexSearcher.<init> Opening Searcher@dbaa63 main
[junit4:junit4] 2> 460955 T1897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@dbaa63 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 460956 T1898 C954 P60000 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index.20130618200050481 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index.20130618200050481;done=true>>]
[junit4:junit4] 2> 460956 T1898 C954 P60000 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index.20130618200050481
[junit4:junit4] 2> 460956 T1898 C954 P60000 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index.20130618200050481
[junit4:junit4] 2> 460956 T1898 C954 P60000 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 460956 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 460957 T1898 C954 P60000 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 460957 T1898 C954 P60000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 460958 T1898 C954 P60000 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 461919 T1824 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 461920 T1824 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "node_name":"127.0.0.1:60000_r",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60000/r"}
[junit4:junit4] 2> 461923 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 461924 T1843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 461924 T1823 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 461924 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 461924 T1830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 461923 T1859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 461927 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 461928 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C957 name=collection1 org.apache.solr.core.SolrCore@19c0d9 url=http://127.0.0.1:40647/r/collection1 node=127.0.0.1:40647_r C957_STATE=coll:control_collection core:collection1 props:{shard=shard1, core=collection1, collection=control_collection, state=active, node_name=127.0.0.1:40647_r, base_url=http://127.0.0.1:40647/r, leader=true}
[junit4:junit4] 2> 461933 T1817 C957 P40647 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 461935 T1817 C957 P40647 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 461936 T1817 C957 P40647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 461949 T1817 C957 P40647 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 461949 T1817 C957 P40647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 461950 T1817 C957 P40647 oass.SolrIndexSearcher.<init> Opening Searcher@47ccf9 main
[junit4:junit4] 2> 461950 T1817 C957 P40647 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 461951 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47ccf9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 461951 T1817 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&commit=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 18
[junit4:junit4] 2> ASYNC NEW_CORE C958 name=collection1 org.apache.solr.core.SolrCore@e8c4fb url=http://127.0.0.1:50641/r/collection1 node=127.0.0.1:50641_r C958_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=active, node_name=127.0.0.1:50641_r, base_url=http://127.0.0.1:50641/r}
[junit4:junit4] 2> 461958 T1855 C958 P50641 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:33880/r/collection1/, StdNode: http://127.0.0.1:50641/r/collection1/, StdNode: http://127.0.0.1:45699/r/collection1/, StdNode: http://127.0.0.1:60000/r/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 461961 T1852 C958 P50641 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 461962 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C955_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=active, node_name=127.0.0.1:45699_r, base_url=http://127.0.0.1:45699/r}
[junit4:junit4] 2> 461965 T1871 C955 P45699 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C954_STATE=coll:collection1 core:collection1 props:{shard=shard1, core=collection1, collection=collection1, state=active, node_name=127.0.0.1:60000_r, base_url=http://127.0.0.1:60000/r}
[junit4:junit4] 2> 461965 T1889 C954 P60000 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 461979 T1836 C956 P33880 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 461979 T1836 C956 P33880 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 461980 T1836 C956 P33880 oass.SolrIndexSearcher.<init> Opening Searcher@3b51b main
[junit4:junit4] 2> 461980 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 461981 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b51b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 461981 T1836 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 19
[junit4:junit4] 2> 461999 T1852 C958 P50641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 461999 T1852 C958 P50641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 461999 T1871 C955 P45699 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 462000 T1871 C955 P45699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 462000 T1889 C954 P60000 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 462001 T1871 C955 P45699 oass.SolrIndexSearcher.<init> Opening Searcher@79beb7 main
[junit4:junit4] 2> 462000 T1852 C958 P50641 oass.SolrIndexSearcher.<init> Opening Searcher@1d27a60 main
[junit4:junit4] 2> 462001 T1871 C955 P45699 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 462001 T1889 C954 P60000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 462002 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79beb7 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 462002 T1861 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceNotFoundException: solr/collection1:id=org.apache.solr.search.FastLRUCache,type=fieldValueCache
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:537)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:138)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 462002 T1852 C958 P50641 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 462003 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d27a60 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 462003 T1889 C954 P60000 oass.SolrIndexSearcher.<init> Opening Searcher@1a68ae2 main
[junit4:junit4] 2> 462003 T1871 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 38
[junit4:junit4] 2> 462004 T1889 C954 P60000 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 462004 T1852 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 43
[junit4:junit4] 2> 462005 T1897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a68ae2 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 462005 T1889 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 40
[junit4:junit4] 2> 462006 T1855 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&commit=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 48
[junit4:junit4] 2> 462007 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 462009 T1839 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=0 status=0 QTime=1
[junit4:junit4] 2> 462010 T1854 C958 P50641 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=0 status=0 QTime=0
[junit4:junit4] 2> 462012 T1872 C955 P45699 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=0 status=0 QTime=1
[junit4:junit4] 2> 462013 T1886 C954 P60000 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=0 status=0 QTime=0
[junit4:junit4] 2> 464017 T1816 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438223122878693376)} 0 1
[junit4:junit4] 2> 464028 T1871 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={update.from=http://127.0.0.1:33880/r/collection1/&_version_=-1438223122883936256&update.distrib=FROMLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438223122883936256)} 0 1
[junit4:junit4] 2> 464028 T1855 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={update.from=http://127.0.0.1:33880/r/collection1/&_version_=-1438223122883936256&update.distrib=FROMLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438223122883936256)} 0 1
[junit4:junit4] 2> 464028 T1889 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={update.from=http://127.0.0.1:33880/r/collection1/&_version_=-1438223122883936256&update.distrib=FROMLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438223122883936256)} 0 1
[junit4:junit4] 2> 464029 T1838 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438223122883936256)} 0 9
[junit4:junit4] 2> 464030 T1872 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 11
[junit4:junit4] 2> 464033 T1817 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {add=[0 (1438223122895470592)]} 0 1
[junit4:junit4] 2> 464041 T1852 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438223122900713472)]} 0 1
[junit4:junit4] 2> 464041 T1869 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438223122900713472)]} 0 1
[junit4:junit4] 2> 464041 T1887 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438223122900713472)]} 0 1
[junit4:junit4] 2> 464042 T1839 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:50641/r/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438223122900713472)]} 0 5
[junit4:junit4] 2> 464042 T1854 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {add=[0]} 0 7
[junit4:junit4] 2> 464045 T1818 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {add=[1 (1438223122908053504)]} 0 1
[junit4:junit4] 2> 464050 T1886 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438223122912247808)]} 0 0
[junit4:junit4] 2> 464051 T1870 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438223122912247808)]} 0 1
[junit4:junit4] 2> 464051 T1853 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438223122912247808)]} 0 1
[junit4:junit4] 2> 464052 T1836 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:60000/r/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438223122912247808)]} 0 4
[junit4:junit4] 2> 464052 T1888 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {add=[1]} 0 6
[junit4:junit4] 2> 464055 T1819 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {add=[2 (1438223122918539264)]} 0 1
[junit4:junit4] 2> 464056 T1838 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:50641/r/collection1/]
[junit4:junit4] 2> 464056 T1838 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50641/r/collection1/ against:[http://127.0.0.1:50641/r/collection1/] result:true
[junit4:junit4] 2> 464057 T1838 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45699/r/collection1/ against:[http://127.0.0.1:50641/r/collection1/] result:false
[junit4:junit4] 2> 464057 T1838 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:60000/r/collection1/ against:[http://127.0.0.1:50641/r/collection1/] result:false
[junit4:junit4] 2> 464060 T1871 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&test.distrib.skip.servers=http://127.0.0.1:50641/r/collection1/&version=2} {add=[2 (1438223122921684992)]} 0 0
[junit4:junit4] 2> 464060 T1889 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&test.distrib.skip.servers=http://127.0.0.1:50641/r/collection1/&version=2} {add=[2 (1438223122921684992)]} 0 0
[junit4:junit4] 2> 464061 T1838 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&test.distrib.skip.servers=http://127.0.0.1:50641/r/collection1/&version=2} {add=[2 (1438223122921684992)]} 0 5
[junit4:junit4] 2> 464063 T1816 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&version=2} {add=[3 (1438223122927976448)]} 0 0
[junit4:junit4] 2> 464066 T1839 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:50641/r/collection1/, http://127.0.0.1:45699/r/collection1/]
[junit4:junit4] 2> 464066 T1839 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50641/r/collection1/ against:[http://127.0.0.1:50641/r/collection1/, http://127.0.0.1:45699/r/collection1/] result:true
[junit4:junit4] 2> 464066 T1839 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45699/r/collection1/ against:[http://127.0.0.1:50641/r/collection1/, http://127.0.0.1:45699/r/collection1/] result:true
[junit4:junit4] 2> 464066 T1839 C956 P33880 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:60000/r/collection1/ against:[http://127.0.0.1:50641/r/collection1/, http://127.0.0.1:45699/r/collection1/] result:false
[junit4:junit4] 2> 464069 T1887 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={distrib.from=http://127.0.0.1:33880/r/collection1/&update.distrib=FROMLEADER&wt=javabin&test.distrib.skip.servers=http://127.0.0.1:50641/r/collection1/&version=2} {add=[3 (1438223122932170752)]} 0 0
[junit4:junit4] 2> 464070 T1839 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={wt=javabin&test.distrib.skip.servers=http://127.0.0.1:50641/r/collection1/&test.distrib.skip.servers=http://127.0.0.1:45699/r/collection1/&version=2} {add=[3 (1438223122932170752)]} 0 5
[junit4:junit4] 2> 464072 T1817 C957 P40647 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464119 T1817 C957 P40647 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 464119 T1817 C957 P40647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 464121 T1817 C957 P40647 oass.SolrIndexSearcher.<init> Opening Searcher@175f386 main
[junit4:junit4] 2> 464121 T1817 C957 P40647 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464122 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@175f386 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 464122 T1817 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&commit=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 50
[junit4:junit4] 2> 464125 T1837 C956 P33880 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:33880/r/collection1/, StdNode: http://127.0.0.1:50641/r/collection1/, StdNode: http://127.0.0.1:45699/r/collection1/, StdNode: http://127.0.0.1:60000/r/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 464126 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464127 T1872 C955 P45699 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464127 T1886 C954 P60000 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464127 T1855 C958 P50641 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464149 T1836 C956 P33880 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 464149 T1836 C956 P33880 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 464151 T1836 C956 P33880 oass.SolrIndexSearcher.<init> Opening Searcher@e03094 main
[junit4:junit4] 2> 464151 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464152 T1845 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e03094 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 464152 T1836 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 26
[junit4:junit4] 2> 464179 T1886 C954 P60000 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 464179 T1886 C954 P60000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 464179 T1872 C955 P45699 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e628e2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 464180 T1872 C955 P45699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 464179 T1855 C958 P50641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cc4fba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 464181 T1855 C958 P50641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 464181 T1886 C954 P60000 oass.SolrIndexSearcher.<init> Opening Searcher@5152fe main
[junit4:junit4] 2> 464182 T1886 C954 P60000 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464182 T1872 C955 P45699 oass.SolrIndexSearcher.<init> Opening Searcher@783c83 main
[junit4:junit4] 2> 464183 T1855 C958 P50641 oass.SolrIndexSearcher.<init> Opening Searcher@38e2d8 main
[junit4:junit4] 2> 464183 T1872 C955 P45699 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464183 T1897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5152fe main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 464184 T1861 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38e2d8 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 464183 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@783c83 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):c3)}
[junit4:junit4] 2> 464183 T1855 C958 P50641 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464185 T1872 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 58
[junit4:junit4] 2> 464185 T1886 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 58
[junit4:junit4] 2> 464185 T1855 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 58
[junit4:junit4] 2> 464186 T1837 C956 P33880 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&commit=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 62
[junit4:junit4] 2> 464187 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 464188 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 464188 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 464190 T1838 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> 464193 T1852 C958 P50641 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=2 status=0 QTime=1
[junit4:junit4] 2> 464195 T1869 C955 P45699 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=3 status=0 QTime=0
[junit4:junit4] 2> 464197 T1888 C954 P60000 oasc.SolrCore.execute [collection1] webapp=/r path=/select params={q=*:*&distrib=false&wt=javabin&tests=checkShardConsistency&rows=0&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> 464199 T1801 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 464204 T1870 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 464204 T1870 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 464208 T1839 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 464208 T1839 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 464208 T1839 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33880/r START replicas=[http://127.0.0.1:50641/r/collection1/, http://127.0.0.1:45699/r/collection1/, http://127.0.0.1:60000/r/collection1/] nUpdates=100
[junit4:junit4] 2> 464210 T1889 C954 P60000 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 464210 T1872 C955 P45699 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 464210 T1853 C958 P50641 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 464210 T1839 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33880/r Received 5 versions from 127.0.0.1:60000/r/collection1/
[junit4:junit4] 2> 464211 T1839 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33880/r Our versions are newer. ourLowThreshold=1438223122883936256 otherHigh=1438223122921684992
[junit4:junit4] 2> 464211 T1839 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33880/r Received 4 versions from 127.0.0.1:45699/r/collection1/
[junit4:junit4] 2> 464211 T1839 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33880/r Our versions are newer. ourLowThreshold=1438223122883936256 otherHigh=1438223122921684992
[junit4:junit4] 2> 464212 T1839 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33880/r Received 3 versions from 127.0.0.1:50641/r/collection1/
[junit4:junit4] 2> 464212 T1839 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33880/r Our versions are newer. ourLowThreshold=1438223122883936256 otherHigh=1438223122912247808
[junit4:junit4] 2> 464212 T1839 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33880/r DONE. sync succeeded
[junit4:junit4] 2> 464212 T1839 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 464213 T1839 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/: try and ask http://127.0.0.1:50641/r/collection1/ to sync
[junit4:junit4] 2> 464213 T1839 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/: try and ask http://127.0.0.1:45699/r/collection1/ to sync
[junit4:junit4] 2> 464213 T1839 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/: try and ask http://127.0.0.1:60000/r/collection1/ to sync
[junit4:junit4] 2> 464214 T1852 C958 P50641 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50641/r START replicas=[http://127.0.0.1:33880/r/collection1/] nUpdates=100
[junit4:junit4] 2> 464214 T1869 C955 P45699 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45699/r START replicas=[http://127.0.0.1:33880/r/collection1/] nUpdates=100
[junit4:junit4] 2> 464214 T1887 C954 P60000 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60000/r START replicas=[http://127.0.0.1:33880/r/collection1/] nUpdates=100
[junit4:junit4] 2> 464215 T1836 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 464216 T1837 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 464216 T1836 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&distrib=false&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 464216 T1887 C954 P60000 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:60000/r Received 5 versions from 127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 464216 T1869 C955 P45699 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45699/r Received 5 versions from 127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 464217 T1887 C954 P60000 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:60000/r Our versions are newer. ourLowThreshold=1438223122883936256 otherHigh=1438223122921684992
[junit4:junit4] 2> 464217 T1852 C958 P50641 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50641/r Received 5 versions from 127.0.0.1:33880/r/collection1/
[junit4:junit4] 2> 464217 T1887 C954 P60000 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60000/r DONE. sync succeeded
[junit4:junit4] 2> 464217 T1869 C955 P45699 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:45699/r Requesting updates from 127.0.0.1:33880/r/collection1/n=1 versions=[1438223122932170752]
[junit4:junit4] 2> 464218 T1887 C954 P60000 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&sync=http://127.0.0.1:33880/r/collection1/&distrib=false&wt=javabin&version=2} status=0 QTime=4
[junit4:junit4] 2> 464218 T1852 C958 P50641 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:50641/r Requesting updates from 127.0.0.1:33880/r/collection1/n=2 versions=[1438223122932170752, 1438223122921684992]
[junit4:junit4] 2> 464219 T1839 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/: sync completed with http://127.0.0.1:60000/r/collection1/
[junit4:junit4] 2> 464220 T1838 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={qt=/get&distrib=false&wt=javabin&version=2&getUpdates=1438223122932170752} status=0 QTime=0
[junit4:junit4] 2> 464220 T1837 C956 P33880 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={qt=/get&distrib=false&wt=javabin&version=2&getUpdates=1438223122932170752,1438223122921684992} status=0 QTime=0
[junit4:junit4] 2> 464222 T1869 C955 P45699 oasup.LogUpdateProcessor.finish [collection1] {add=[3 (1438223122932170752)]} 0 1
[junit4:junit4] 2> 464223 T1869 C955 P45699 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45699/r DONE. sync succeeded
[junit4:junit4] 2> 464223 T1852 C958 P50641 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438223122921684992), 3 (1438223122932170752)]} 0 2
[junit4:junit4] 2> 464223 T1869 C955 P45699 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&sync=http://127.0.0.1:33880/r/collection1/&distrib=false&wt=javabin&version=2} status=0 QTime=9
[junit4:junit4] 2> 464223 T1852 C958 P50641 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50641/r DONE. sync succeeded
[junit4:junit4] 2> 464224 T1852 C958 P50641 oasc.SolrCore.execute [collection1] webapp=/r path=/get params={getVersions=100&qt=/get&sync=http://127.0.0.1:33880/r/collection1/&distrib=false&wt=javabin&version=2} status=0 QTime=10
[junit4:junit4] 2> 464224 T1839 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/: sync completed with http://127.0.0.1:45699/r/collection1/
[junit4:junit4] 2> 464224 T1839 oasc.SyncStrategy.syncToMe http://127.0.0.1:33880/r/collection1/: sync completed with http://127.0.0.1:50641/r/collection1/
[junit4:junit4] 2> 464225 T1839 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=17
[junit4:junit4] 2> 464225 T1870 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=22
[junit4:junit4] 2> 464225 T1801 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 464226 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 464226 T1801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 464227 T1801 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 464229 T1819 C957 P40647 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464279 T1819 C957 P40647 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@245e05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 464279 T1819 C957 P40647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 464280 T1819 C957 P40647 oass.SolrIndexSearcher.<init> Opening Searcher@1ee6f3 main
[junit4:junit4] 2> 464281 T1819 C957 P40647 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464281 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ee6f3 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 464282 T1819 C957 P40647 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={softCommit=false&commit=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 53
[junit4:junit4] 2> 464284 T1871 C955 P45699 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:33880/r/collection1/, StdNode: http://127.0.0.1:50641/r/collection1/, StdNode: http://127.0.0.1:45699/r/collection1/, StdNode: http://127.0.0.1:60000/r/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 464285 T1854 C958 P50641 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464286 T1886 C954 P60000 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464286 T1869 C955 P45699 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464292 T1836 C956 P33880 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 464309 T1886 C954 P60000 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b88852; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_7,generation=7}
[junit4:junit4] 2> 464309 T1886 C954 P60000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 464310 T1886 C954 P60000 oass.SolrIndexSearcher.<init> Opening Searcher@23b2be main
[junit4:junit4] 2> 464310 T1886 C954 P60000 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 464311 T1897 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23b2be main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 464312 T1886 C954 P60000 oasup.LogUpdateProcessor.finish [collection1] webapp=/r path=/update params={expungeDeletes=false&softCommit=false&commit=true&commit_end_point=true&wt=javabin&waitSearcher=true&version=2} {commit=} 0 26
[junit4:junit4] 2> 464319 T1836 C956 P33880 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371596432910/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cac28e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTe
[...truncated too long message...]
/solr/collections/collection1/leader_elect/shard1/election/89888943827320842-4-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888943827320846-2-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888943827320844-5-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:40647_r",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40647/r",
[junit4:junit4] 1> "leader":"true"}}}}},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "2":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:33880_r",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:33880/r"},
[junit4:junit4] 1> "3":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "node_name":"127.0.0.1:50641_r",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50641/r"},
[junit4:junit4] 1> "4":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:45699_r",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45699/r",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "5":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:60000_r",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60000/r"}}}}}}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89888943827320844-127.0.0.1:60000_r-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888943827320842-127.0.0.1:45699_r-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888943827320835-127.0.0.1:40647_r-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888943827320846-127.0.0.1:33880_r-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89888943827320835-127.0.0.1:40647_r-n_0000000000"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=14FA98A28E8105C6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ja_JP -Dtests.timezone=America/Jujuy -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 98.8s J1 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:33880/r/collection1lastClient and got 266 from http://127.0.0.1:45699/r/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([14FA98A28E8105C6:951C16BAF9DE65FA]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 542128 T1801 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 98762 T1800 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 542233 T1876 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 542233 T1876 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 542233 T1876 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 543324 T1893 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 543325 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 543325 T1893 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=136), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=136)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ja_JP, timezone=America/Jujuy
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=3,free=81841976,total=301989888
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestCollationField, ChaosMonkeySafeLeaderTest, DistanceFunctionTest, TestDFRSimilarityFactory, CopyFieldTest, DebugComponentTest, SuggesterWFSTTest, UpdateParamsTest, OutputWriterTest, TestManagedSchema, TestLuceneMatchVersion, FileUtilsTest, TestIndexSearcher, TestSolrJ, IndexBasedSpellCheckerTest, NumericFieldsTest, TimeZoneUtilsTest, SolrCmdDistributorTest, CachingDirectoryFactoryTest, SampleTest, HardAutoCommitTest, TestSchemaVersionResource, TestUpdate, URLClassifyProcessorTest, StandardRequestHandlerTest, TestAtomicUpdateErrorCases, IndexReaderFactoryTest, TestSolrXml, TestBM25SimilarityFactory, BadCopyFieldTest, TestUtils, TestFieldResource, TestSuggestSpellingConverter, TestHashPartitioner, EchoParamsTest, TestLMJelinekMercerSimilarityFactory, MultiTermTest, TestDistributedGrouping, PathHierarchyTokenizerFactoryTest, CurrencyFieldXmlFileTest, TestPartialUpdateDeduplication, BadIndexSchemaTest, NoCacheHeaderTest, DisMaxRequestHandlerTest, TestMergePolicyConfig, FullSolrCloudDistribCmdsTest, TestClassNameShortening, TestRTGBase, SortByFunctionTest, ZkNodePropsTest, TestStressReorder, TestReloadAndDeleteDocs, TestJmxMonitoredMap, TestDistributedSearch, RequestHandlersTest, TestCodecSupport, StatsComponentTest, ShardRoutingCustomTest, DirectUpdateHandlerTest, SolrTestCaseJ4Test, RequiredFieldsTest, TestComponentsName, UniqFieldsUpdateProcessorFactoryTest, TestShardHandlerFactory, JSONWriterTest, TestNumberUtils, TestCloudManagedSchemaAddField, TestRandomFaceting, TestCSVLoader, HighlighterConfigTest, TestUniqueKeyFieldResource, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J1 in 100.04s, 1 test, 1 failure <<< FAILURES!
[...truncated 499 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1252 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 36 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
6198 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6198/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
2 tests failed.
REGRESSION: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:43159/_/onenodecollectioncore returned non ok status:404, message:Can not find: /_/onenodecollectioncore/update
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:43159/_/onenodecollectioncore returned non ok status:404, message:Can not find: /_/onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([34FFADCFCB0F3CD9:B51923D7BC505CE5]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:46520/collection1lastClient and got 249 from http://127.0.0.1:46043/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:46520/collection1lastClient and got 249 from http://127.0.0.1:46043/collection1
at __randomizedtesting.SeedInfo.seed([34FFADCFCB0F3CD9:B51923D7BC505CE5]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
Build Log:
[...truncated 9268 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> 130445 T367 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/
[junit4:junit4] 2> 130448 T367 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668
[junit4:junit4] 2> 130449 T367 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 130450 T368 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 130550 T367 oasc.ZkTestServer.run start zk server on port:45859
[junit4:junit4] 2> 130552 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130583 T374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f319a26f name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130583 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130584 T367 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 130590 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130591 T376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@439eedaf name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130592 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130592 T367 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 130595 T367 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 130597 T367 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 130599 T367 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 130603 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 130603 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 130609 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 130610 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 130614 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 130615 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 130618 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 130619 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 130621 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 130622 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 130625 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 130626 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 130629 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 130629 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 130632 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 130633 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 130636 T367 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 130637 T367 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 130745 T367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 130750 T367 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50784
[junit4:junit4] 2> 130750 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 130751 T367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 130752 T367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860
[junit4:junit4] 2> 130752 T367 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/solr.xml
[junit4:junit4] 2> 130753 T367 oasc.CoreContainer.<init> New CoreContainer -1120475356
[junit4:junit4] 2> 130753 T367 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/'
[junit4:junit4] 2> 130754 T367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/'
[junit4:junit4] 2> 130802 T367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 130803 T367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 130803 T367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 130804 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 130805 T367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 130805 T367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 130806 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 130807 T367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 130807 T367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 130808 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 130813 T367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 130814 T367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45859/solr
[junit4:junit4] 2> 130814 T367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 130815 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130819 T387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fce6b354 name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130819 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130821 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 130827 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130829 T389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@113dc498 name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130829 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130831 T367 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 130834 T367 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 130837 T367 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 130839 T367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50784__
[junit4:junit4] 2> 130840 T367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50784__
[junit4:junit4] 2> 130844 T367 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 130850 T367 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 130852 T367 oasc.Overseer.start Overseer (id=89888612199366659-127.0.0.1:50784__-n_0000000000) starting
[junit4:junit4] 2> 130856 T367 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 130859 T391 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 130860 T367 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 130862 T367 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 130864 T367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 130866 T390 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 130868 T392 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 130869 T392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 132369 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 132370 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50784/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50784__",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 132370 T390 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 132371 T390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 132375 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 132870 T392 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/collection1
[junit4:junit4] 2> 132871 T392 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 132872 T392 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 132872 T392 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 132873 T392 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/collection1/'
[junit4:junit4] 2> 132874 T392 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/collection1/lib/README' to classloader
[junit4:junit4] 2> 132874 T392 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 132904 T392 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 132934 T392 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 132936 T392 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 132940 T392 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 133241 T392 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 133247 T392 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 133250 T392 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 133265 T392 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 133271 T392 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 133277 T392 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 133279 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 133279 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 133279 T392 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 133280 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 133281 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 133281 T392 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 133282 T392 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1371591372860/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/
[junit4:junit4] 2> 133282 T392 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@da386b55
[junit4:junit4] 2> 133282 T392 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 133283 T392 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data
[junit4:junit4] 2> 133284 T392 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/index/
[junit4:junit4] 2> 133284 T392 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 133285 T392 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/index
[junit4:junit4] 2> 133287 T392 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@11658b42 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ff77a7e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 133288 T392 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 133290 T392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 133291 T392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 133291 T392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 133292 T392 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 133294 T392 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 133294 T392 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 133296 T392 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 133297 T392 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 133298 T392 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 133300 T392 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 133304 T392 oass.SolrIndexSearcher.<init> Opening Searcher@5a2fe52e main
[junit4:junit4] 2> 133306 T392 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/tlog
[junit4:junit4] 2> 133308 T392 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 133309 T392 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 133314 T393 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a2fe52e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 133315 T392 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 133315 T392 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50784/_ collection:control_collection shard:shard1
[junit4:junit4] 2> 133318 T392 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 133325 T392 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 133332 T392 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 133336 T392 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 133336 T392 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 133338 T392 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50784/_/collection1/
[junit4:junit4] 2> 133338 T392 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 133339 T392 oasc.SyncStrategy.syncToMe http://127.0.0.1:50784/_/collection1/ has no replicas
[junit4:junit4] 2> 133339 T392 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50784/_/collection1/
[junit4:junit4] 2> 133340 T392 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 133879 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 133887 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 133898 T392 oasc.ZkController.register We are http://127.0.0.1:50784/_/collection1/ and leader is http://127.0.0.1:50784/_/collection1/
[junit4:junit4] 2> 133899 T392 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50784/_
[junit4:junit4] 2> 133899 T392 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 133899 T392 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 133899 T392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 133901 T392 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 133903 T367 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 133904 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 133905 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 133913 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 133916 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 133917 T396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78f790d5 name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 133918 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 133919 T367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 133923 T367 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 134017 T367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 134019 T367 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46951
[junit4:junit4] 2> 134020 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 134021 T367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 134021 T367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143
[junit4:junit4] 2> 134022 T367 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/solr.xml
[junit4:junit4] 2> 134022 T367 oasc.CoreContainer.<init> New CoreContainer -1176669112
[junit4:junit4] 2> 134023 T367 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/'
[junit4:junit4] 2> 134024 T367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/'
[junit4:junit4] 2> 134069 T367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 134070 T367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 134071 T367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 134071 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 134072 T367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 134073 T367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 134074 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 134074 T367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 134075 T367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 134076 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 134083 T367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 134084 T367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45859/solr
[junit4:junit4] 2> 134084 T367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 134085 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 134087 T407 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd30e7d4 name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 134087 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 134089 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 134095 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 134097 T409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a02d7e4 name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 134097 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 134101 T367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 135104 T367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46951__
[junit4:junit4] 2> 135106 T367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46951__
[junit4:junit4] 2> 135109 T389 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 135109 T409 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 135109 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 135109 T396 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 135115 T410 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 135115 T410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 135392 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 135393 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50784/_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50784__",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 135397 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 135397 T390 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 135397 T390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 135532 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 135532 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 135532 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 136117 T410 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/collection1
[junit4:junit4] 2> 136117 T410 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 136118 T410 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 136119 T410 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 136120 T410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/collection1/'
[junit4:junit4] 2> 136121 T410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/collection1/lib/README' to classloader
[junit4:junit4] 2> 136122 T410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 136160 T410 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 136206 T410 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 136208 T410 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 136214 T410 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 136545 T410 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 136550 T410 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 136553 T410 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 136565 T410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 136569 T410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 136573 T410 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 136574 T410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 136575 T410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 136575 T410 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 136576 T410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 136577 T410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 136577 T410 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 136577 T410 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/
[junit4:junit4] 2> 136578 T410 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@da386b55
[junit4:junit4] 2> 136578 T410 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 136579 T410 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1
[junit4:junit4] 2> 136579 T410 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/index/
[junit4:junit4] 2> 136579 T410 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 136580 T410 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/index
[junit4:junit4] 2> 136581 T410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@776bd81f lockFactory=org.apache.lucene.store.NativeFSLockFactory@82fad443),segFN=segments_1,generation=1}
[junit4:junit4] 2> 136582 T410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 136583 T410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 136584 T410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 136584 T410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 136585 T410 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 136585 T410 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 136586 T410 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 136586 T410 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 136587 T410 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 136587 T410 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 136588 T410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 136590 T410 oass.SolrIndexSearcher.<init> Opening Searcher@8880084d main
[junit4:junit4] 2> 136590 T410 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/tlog
[junit4:junit4] 2> 136591 T410 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 136591 T410 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 136595 T411 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8880084d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 136597 T410 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 136597 T410 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46951/_ collection:collection1 shard:shard1
[junit4:junit4] 2> 136598 T410 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 136603 T410 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 136605 T410 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 136605 T410 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 136606 T410 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46951/_/collection1/
[junit4:junit4] 2> 136606 T410 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 136606 T410 oasc.SyncStrategy.syncToMe http://127.0.0.1:46951/_/collection1/ has no replicas
[junit4:junit4] 2> 136607 T410 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46951/_/collection1/
[junit4:junit4] 2> 136607 T410 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 137040 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 137046 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 137047 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 137046 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 137063 T410 oasc.ZkController.register We are http://127.0.0.1:46951/_/collection1/ and leader is http://127.0.0.1:46951/_/collection1/
[junit4:junit4] 2> 137064 T410 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46951/_
[junit4:junit4] 2> 137064 T410 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 137064 T410 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 137064 T410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 137066 T410 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 137068 T367 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 137068 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 137069 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 137149 T367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 137151 T367 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43159
[junit4:junit4] 2> 137152 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 137153 T367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 137153 T367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292
[junit4:junit4] 2> 137154 T367 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/solr.xml
[junit4:junit4] 2> 137155 T367 oasc.CoreContainer.<init> New CoreContainer 241613837
[junit4:junit4] 2> 137155 T367 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/'
[junit4:junit4] 2> 137156 T367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/'
[junit4:junit4] 2> 137193 T367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 137194 T367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 137195 T367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 137195 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 137196 T367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 137196 T367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 137197 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 137198 T367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 137198 T367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 137199 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 137203 T367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 137204 T367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45859/solr
[junit4:junit4] 2> 137204 T367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 137206 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 137208 T423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8139200b name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 137208 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 137210 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 137215 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 137216 T425 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@80b38190 name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 137217 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 137220 T367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 138224 T367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43159__
[junit4:junit4] 2> 138225 T367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43159__
[junit4:junit4] 2> 138227 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 138228 T409 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 138228 T425 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 138228 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 138228 T389 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 138229 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 138229 T396 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 138233 T426 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 138234 T426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 138552 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 138553 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 138556 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43159/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:43159__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 138556 T390 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 138557 T390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 138559 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 138560 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 138560 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 138560 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 139235 T426 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/collection1
[junit4:junit4] 2> 139235 T426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 139236 T426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 139237 T426 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 139238 T426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/collection1/'
[junit4:junit4] 2> 139239 T426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/collection1/lib/README' to classloader
[junit4:junit4] 2> 139239 T426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 139278 T426 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 139315 T426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 139317 T426 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 139321 T426 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 139648 T426 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 139653 T426 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 139656 T426 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 139668 T426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 139672 T426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 139675 T426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 139677 T426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 139677 T426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 139677 T426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 139678 T426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 139679 T426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 139679 T426 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 139679 T426 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1371591379292/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/
[junit4:junit4] 2> 139680 T426 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@da386b55
[junit4:junit4] 2> 139680 T426 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 139681 T426 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2
[junit4:junit4] 2> 139681 T426 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/index/
[junit4:junit4] 2> 139681 T426 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 139682 T426 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/index
[junit4:junit4] 2> 139683 T426 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@70b4a8e6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@83c125a2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 139683 T426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 139685 T426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 139685 T426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 139686 T426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 139687 T426 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 139687 T426 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 139687 T426 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 139688 T426 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 139688 T426 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 139689 T426 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 139690 T426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 139692 T426 oass.SolrIndexSearcher.<init> Opening Searcher@91d4bcc1 main
[junit4:junit4] 2> 139692 T426 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/tlog
[junit4:junit4] 2> 139693 T426 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 139693 T426 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 139698 T427 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@91d4bcc1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 139700 T426 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 139700 T426 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43159/_ collection:collection1 shard:shard2
[junit4:junit4] 2> 139701 T426 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 139706 T426 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 139708 T426 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 139709 T426 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 139709 T426 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43159/_/collection1/
[junit4:junit4] 2> 139709 T426 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 139710 T426 oasc.SyncStrategy.syncToMe http://127.0.0.1:43159/_/collection1/ has no replicas
[junit4:junit4] 2> 139710 T426 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43159/_/collection1/
[junit4:junit4] 2> 139710 T426 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 140065 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 140072 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 140072 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 140072 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 140074 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 140116 T426 oasc.ZkController.register We are http://127.0.0.1:43159/_/collection1/ and leader is http://127.0.0.1:43159/_/collection1/
[junit4:junit4] 2> 140116 T426 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43159/_
[junit4:junit4] 2> 140116 T426 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 140117 T426 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 140117 T426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 140119 T426 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 140121 T367 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 140121 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 140122 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 140202 T367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 140205 T367 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60467
[junit4:junit4] 2> 140206 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 140206 T367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 140207 T367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345
[junit4:junit4] 2> 140207 T367 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/solr.xml
[junit4:junit4] 2> 140208 T367 oasc.CoreContainer.<init> New CoreContainer -1795795558
[junit4:junit4] 2> 140208 T367 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/'
[junit4:junit4] 2> 140209 T367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/'
[junit4:junit4] 2> 140246 T367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 140247 T367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 140247 T367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 140248 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 140249 T367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 140249 T367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 140250 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 140250 T367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 140251 T367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 140251 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 140255 T367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 140256 T367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45859/solr
[junit4:junit4] 2> 140257 T367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 140258 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 140259 T439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8be7546f name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 140260 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 140261 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 140266 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 140267 T441 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f0f0b81e name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 140268 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 140271 T367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 141274 T367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60467__
[junit4:junit4] 2> 141276 T367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60467__
[junit4:junit4] 2> 141279 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 141279 T409 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 141279 T425 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 141279 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141280 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141281 T389 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 141282 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141281 T441 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 141282 T396 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 141290 T442 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 141290 T442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 141578 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 141579 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43159/_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:43159__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 141582 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60467/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:60467__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 141582 T390 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 141583 T390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 141587 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141587 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141587 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141587 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 141587 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 142292 T442 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/collection1
[junit4:junit4] 2> 142292 T442 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 142293 T442 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 142293 T442 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 142294 T442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/collection1/'
[junit4:junit4] 2> 142295 T442 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/collection1/lib/README' to classloader
[junit4:junit4] 2> 142295 T442 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 142328 T442 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 142357 T442 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 142358 T442 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 142362 T442 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 142663 T442 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 142668 T442 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 142671 T442 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 142683 T442 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 142687 T442 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 142691 T442 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 142692 T442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 142692 T442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 142693 T442 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 142694 T442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 142694 T442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 142694 T442 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 142695 T442 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1371591382345/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/
[junit4:junit4] 2> 142695 T442 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@da386b55
[junit4:junit4] 2> 142696 T442 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 142696 T442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3
[junit4:junit4] 2> 142697 T442 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index/
[junit4:junit4] 2> 142697 T442 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 142697 T442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index
[junit4:junit4] 2> 142699 T442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e40085f8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b365c55d),segFN=segments_1,generation=1}
[junit4:junit4] 2> 142699 T442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 142701 T442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 142701 T442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 142702 T442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 142702 T442 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 142703 T442 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 142703 T442 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 142704 T442 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 142704 T442 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 142705 T442 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 142706 T442 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 142708 T442 oass.SolrIndexSearcher.<init> Opening Searcher@ae5500d6 main
[junit4:junit4] 2> 142708 T442 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/tlog
[junit4:junit4] 2> 142709 T442 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 142709 T442 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 142713 T443 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ae5500d6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 142715 T442 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 142716 T442 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60467/_ collection:collection1 shard:shard1
[junit4:junit4] 2> 142718 T442 oasc.ZkController.register We are http://127.0.0.1:60467/_/collection1/ and leader is http://127.0.0.1:46951/_/collection1/
[junit4:junit4] 2> 142718 T442 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60467/_
[junit4:junit4] 2> 142718 T442 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 142719 T442 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C199 name=collection1 org.apache.solr.core.SolrCore@f472a525 url=http://127.0.0.1:60467/_/collection1 node=127.0.0.1:60467__ C199_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:60467/_, state=down, node_name=127.0.0.1:60467__, collection=collection1, core=collection1}
[junit4:junit4] 2> 142721 T444 C199 P60467 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 142721 T442 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 142721 T444 C199 P60467 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 142722 T444 C199 P60467 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 142722 T444 C199 P60467 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 142722 T367 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 142723 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 142723 T444 C199 P60467 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 142724 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 142733 T404 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 142857 T367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 142860 T367 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52410
[junit4:junit4] 2> 142861 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 142862 T367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 142863 T367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949
[junit4:junit4] 2> 142864 T367 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/solr.xml
[junit4:junit4] 2> 142865 T367 oasc.CoreContainer.<init> New CoreContainer 2013843607
[junit4:junit4] 2> 142866 T367 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/'
[junit4:junit4] 2> 142867 T367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/'
[junit4:junit4] 2> 142925 T367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 142926 T367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 142926 T367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 142927 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 142928 T367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 142929 T367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 142929 T367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 142930 T367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 142931 T367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 142932 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 142938 T367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 142939 T367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45859/solr
[junit4:junit4] 2> 142940 T367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 142941 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 142943 T456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a279a0d name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 142943 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 142945 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 142950 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 142951 T458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8f19330 name:ZooKeeperConnection Watcher:127.0.0.1:45859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 142951 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 142954 T367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 143095 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 143096 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60467/_",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:60467__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 143100 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143100 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143100 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143100 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143100 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143100 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143733 T404 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 143734 T404 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=4&wt=javabin&nodeName=127.0.0.1:60467__&core=collection1} status=0 QTime=1002
[junit4:junit4] 2> 143957 T367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52410__
[junit4:junit4] 2> 143960 T367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52410__
[junit4:junit4] 2> 143962 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143962 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143962 T389 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 143962 T409 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 143962 T425 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 143964 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 143964 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 143964 T396 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 143963 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 143962 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 143965 T441 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 143969 T458 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 143971 T459 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 143972 T459 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 144604 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 144605 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52410/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "node_name":"127.0.0.1:52410__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 144606 T390 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 144606 T390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 144610 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 144610 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 144610 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 144610 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 144610 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 144610 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 144974 T459 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/collection1
[junit4:junit4] 2> 144974 T459 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 144976 T459 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 144976 T459 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 144979 T459 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/collection1/'
[junit4:junit4] 2> 144980 T459 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/collection1/lib/README' to classloader
[junit4:junit4] 2> 144981 T459 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 145005 T459 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 145051 T459 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 145052 T459 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 145057 T459 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 145380 T459 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 145386 T459 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 145389 T459 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 145400 T459 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 145404 T459 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 145408 T459 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 145409 T459 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 145409 T459 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 145409 T459 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 145410 T459 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 145411 T459 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 145411 T459 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 145411 T459 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1371591384949/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/
[junit4:junit4] 2> 145412 T459 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@da386b55
[junit4:junit4] 2> 145412 T459 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 145413 T459 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4
[junit4:junit4] 2> 145413 T459 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index/
[junit4:junit4] 2> 145414 T459 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 145414 T459 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index
[junit4:junit4] 2> 145416 T459 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ca6b5ce0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a7b682d),segFN=segments_1,generation=1}
[junit4:junit4] 2> 145416 T459 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 145418 T459 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 145418 T459 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 145419 T459 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 145420 T459 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 145420 T459 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 145420 T459 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 145421 T459 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 145421 T459 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 145422 T459 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 145423 T459 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 145425 T459 oass.SolrIndexSearcher.<init> Opening Searcher@fb1fe383 main
[junit4:junit4] 2> 145425 T459 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/tlog
[junit4:junit4] 2> 145426 T459 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 145426 T459 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 145432 T460 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fb1fe383 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 145435 T459 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 145435 T459 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52410/_ collection:collection1 shard:shard2
[junit4:junit4] 2> 145438 T459 oasc.ZkController.register We are http://127.0.0.1:52410/_/collection1/ and leader is http://127.0.0.1:43159/_/collection1/
[junit4:junit4] 2> 145439 T459 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52410/_
[junit4:junit4] 2> 145440 T459 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 145441 T459 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C200 name=collection1 org.apache.solr.core.SolrCore@558fdb31 url=http://127.0.0.1:52410/_/collection1 node=127.0.0.1:52410__ C200_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:52410/_, state=down, node_name=127.0.0.1:52410__, collection=collection1, core=collection1}
[junit4:junit4] 2> 145442 T461 C200 P52410 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 145442 T461 C200 P52410 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 145442 T459 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 145442 T461 C200 P52410 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 145443 T461 C200 P52410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 145444 T367 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 145444 T367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 145444 T461 C200 P52410 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 145445 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 145450 T367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 145451 T418 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 145452 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 145459 T402 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> ASYNC NEW_CORE C201 name=collection1 org.apache.solr.core.SolrCore@f472a525 url=http://127.0.0.1:60467/_/collection1 node=127.0.0.1:60467__ C201_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:60467/_, state=recovering, node_name=127.0.0.1:60467__, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 145737 T444 C201 P60467 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46951/_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 145740 T444 C201 P60467 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 145746 T444 C201 P60467 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60467/_ START replicas=[http://127.0.0.1:46951/_/collection1/] nUpdates=100
[junit4:junit4] 2> 145748 T444 C201 P60467 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 145748 T444 C201 P60467 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 145749 T444 C201 P60467 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 145749 T444 C201 P60467 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 145749 T444 C201 P60467 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 145750 T444 C201 P60467 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46951/_/collection1/. core=collection1
[junit4:junit4] 2> 145750 T444 C201 P60467 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C202 name=collection1 org.apache.solr.core.SolrCore@ab15e19 url=http://127.0.0.1:46951/_/collection1 node=127.0.0.1:46951__ C202_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:46951/_, state=active, node_name=127.0.0.1:46951__, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 145756 T404 C202 P46951 oasc.SolrCore.execute [collection1] webapp=/_ path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 145761 T403 C202 P46951 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 145762 T403 C202 P46951 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@776bd81f lockFactory=org.apache.lucene.store.NativeFSLockFactory@82fad443),segFN=segments_1,generation=1}
[junit4:junit4] 2> 145763 T403 C202 P46951 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 145764 T403 C202 P46951 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@776bd81f lockFactory=org.apache.lucene.store.NativeFSLockFactory@82fad443),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@776bd81f lockFactory=org.apache.lucene.store.NativeFSLockFactory@82fad443),segFN=segments_2,generation=2}
[junit4:junit4] 2> 145764 T403 C202 P46951 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 145765 T403 C202 P46951 oass.SolrIndexSearcher.<init> Opening Searcher@602140ba realtime
[junit4:junit4] 2> 145765 T403 C202 P46951 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 145766 T403 C202 P46951 oasup.LogUpdateProcessor.finish [collection1] webapp=/_ path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 10
[junit4:junit4] 2> 145776 T444 C201 P60467 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 145776 T444 C201 P60467 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 145780 T444 C201 P60467 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 145791 T403 C202 P46951 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 145792 T403 C202 P46951 oasc.SolrCore.execute [collection1] webapp=/_ path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=10
[junit4:junit4] 2> 145793 T444 C201 P60467 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 145793 T444 C201 P60467 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 145793 T444 C201 P60467 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 145796 T405 C202 P46951 oasc.SolrCore.execute [collection1] webapp=/_ path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 145797 T444 C201 P60467 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 145797 T444 C201 P60467 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index.20130618133628016
[junit4:junit4] 2> 145798 T444 C201 P60467 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@867930cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e4be34b) fullCopy=false
[junit4:junit4] 2> 145801 T404 C202 P46951 oasc.SolrCore.execute [collection1] webapp=/_ path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 145803 T444 C201 P60467 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 145804 T444 C201 P60467 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 145804 T444 C201 P60467 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 145805 T444 C201 P60467 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e40085f8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b365c55d),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e40085f8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b365c55d),segFN=segments_2,generation=2}
[junit4:junit4] 2> 145806 T444 C201 P60467 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 145806 T444 C201 P60467 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 145807 T444 C201 P60467 oass.SolrIndexSearcher.<init> Opening Searcher@1391d19e main
[junit4:junit4] 2> 145807 T443 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1391d19e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 145808 T444 C201 P60467 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index.20130618133628016 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index.20130618133628016;done=true>>]
[junit4:junit4] 2> 145808 T444 C201 P60467 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index.20130618133628016
[junit4:junit4] 2> 145809 T444 C201 P60467 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index.20130618133628016
[junit4:junit4] 2> 145809 T444 C201 P60467 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 145809 T444 C201 P60467 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 145809 T444 C201 P60467 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 145809 T444 C201 P60467 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 145810 T444 C201 P60467 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 146117 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 146118 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52410/_",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "node_name":"127.0.0.1:52410__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 146121 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 146121 T390 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 146121 T390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 146124 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60467/_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:60467__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 146127 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146128 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146127 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146128 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146127 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146128 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146452 T418 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 146453 T418 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=5&wt=javabin&nodeName=127.0.0.1:52410__&core=collection1} status=0 QTime=1002
[junit4:junit4] 2> 146460 T402 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/onenodecollectioncore
[junit4:junit4] 2> 146461 T402 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 146462 T402 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 146463 T402 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 146464 T402 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 146465 T402 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 146468 T402 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 146469 T402 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/onenodecollectioncore/'
[junit4:junit4] 2> 146503 T402 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 146533 T402 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 146534 T402 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 146538 T402 oass.IndexSchema.readSchema [onenodecollectioncore] Schema name=test
[junit4:junit4] 2> 146867 T402 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 146872 T402 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 146875 T402 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 146887 T402 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 146891 T402 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 146895 T402 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 146896 T402 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 146896 T402 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 146897 T402 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 146898 T402 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 146898 T402 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 146899 T402 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 146899 T402 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1371591376143/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/
[junit4:junit4] 2> 146900 T402 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@da386b55
[junit4:junit4] 2> 146900 T402 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 146901 T402 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection
[junit4:junit4] 2> 146902 T402 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/index/
[junit4:junit4] 2> 146902 T402 oasc.SolrCore.initIndex WARN [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 146903 T402 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/index
[junit4:junit4] 2> 146904 T402 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6a95f9d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@fa817b42),segFN=segments_1,generation=1}
[junit4:junit4] 2> 146905 T402 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 146906 T402 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 146907 T402 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 146908 T402 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 146908 T402 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 146909 T402 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 146909 T402 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 146910 T402 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 146910 T402 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 146911 T402 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 146912 T402 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 146914 T402 oass.SolrIndexSearcher.<init> Opening Searcher@ad00924d main
[junit4:junit4] 2> 146915 T402 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/tlog
[junit4:junit4] 2> 146916 T402 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 146916 T402 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 146920 T465 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@ad00924d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 146922 T402 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 146922 T402 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:46951/_ collection:onenodecollection shard:shard1
[junit4:junit4] 2> 146924 T402 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 146929 T402 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 146932 T402 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 146933 T402 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 146934 T402 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46951/_/onenodecollectioncore/
[junit4:junit4] 2> 146934 T402 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 146935 T402 oasc.SyncStrategy.syncToMe http://127.0.0.1:46951/_/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 146936 T402 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46951/_/onenodecollectioncore/
[junit4:junit4] 2> 146937 T402 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> 147634 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 147640 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 147640 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 147640 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 147640 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 147640 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 147640 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 147644 T402 oasc.ZkController.register We are http://127.0.0.1:46951/_/onenodecollectioncore/ and leader is http://127.0.0.1:46951/_/onenodecollectioncore/
[junit4:junit4] 2> 147644 T402 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:46951/_
[junit4:junit4] 2> 147645 T402 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 147645 T402 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 147647 T402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 147648 T402 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection&action=CREATE&version=2&numShards=1&name=onenodecollectioncore&wt=javabin&roles=none&collection=onenodecollection} status=0 QTime=2189
[junit4:junit4] 2> 147649 T367 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 147649 T367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C203 name=collection1 org.apache.solr.core.SolrCore@558fdb31 url=http://127.0.0.1:52410/_/collection1 node=127.0.0.1:52410__ C203_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:52410/_, state=recovering, node_name=127.0.0.1:52410__, collection=collection1, shard=shard2, core=collection1}
[junit4:junit4] 2> 148454 T461 C203 P52410 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43159/_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 148454 T461 C203 P52410 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52410/_ START replicas=[http://127.0.0.1:43159/_/collection1/] nUpdates=100
[junit4:junit4] 2> 148454 T461 C203 P52410 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 148455 T461 C203 P52410 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 148455 T461 C203 P52410 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 148455 T461 C203 P52410 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 148455 T461 C203 P52410 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 148456 T461 C203 P52410 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43159/_/collection1/. core=collection1
[junit4:junit4] 2> 148456 T461 C203 P52410 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C204 name=collection1 org.apache.solr.core.SolrCore@17a44e70 url=http://127.0.0.1:43159/_/collection1 node=127.0.0.1:43159__ C204_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:43159/_, state=active, node_name=127.0.0.1:43159__, collection=collection1, shard=shard2, core=collection1, leader=true}
[junit4:junit4] 2> 148460 T419 C204 P43159 oasc.SolrCore.execute [collection1] webapp=/_ path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 148461 T421 C204 P43159 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148463 T421 C204 P43159 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@70b4a8e6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@83c125a2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 148463 T421 C204 P43159 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 148464 T421 C204 P43159 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@70b4a8e6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@83c125a2),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@70b4a8e6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@83c125a2),segFN=segments_2,generation=2}
[junit4:junit4] 2> 148465 T421 C204 P43159 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 148465 T421 C204 P43159 oass.SolrIndexSearcher.<init> Opening Searcher@eca501cc realtime
[junit4:junit4] 2> 148466 T421 C204 P43159 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148467 T421 C204 P43159 oasup.LogUpdateProcessor.finish [collection1] webapp=/_ path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 6
[junit4:junit4] 2> 148468 T461 C203 P52410 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 148468 T461 C203 P52410 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 148470 T419 C204 P43159 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 148470 T419 C204 P43159 oasc.SolrCore.execute [collection1] webapp=/_ path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=1
[junit4:junit4] 2> 148471 T461 C203 P52410 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 148471 T461 C203 P52410 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 148472 T461 C203 P52410 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 148474 T421 C204 P43159 oasc.SolrCore.execute [collection1] webapp=/_ path=/replication params={qt=/replication&version=2&generation=2&command=filelist&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 148475 T461 C203 P52410 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 148476 T461 C203 P52410 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index.20130618133630694
[junit4:junit4] 2> 148476 T461 C203 P52410 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@5b5d5f93 lockFactory=org.apache.lucene.store.NativeFSLockFactory@704495c3) fullCopy=false
[junit4:junit4] 2> 148478 T418 C204 P43159 oasc.SolrCore.execute [collection1] webapp=/_ path=/replication params={checksum=true&qt=/replication&generation=2&command=filecontent&file=segments_2&wt=filestream} status=0 QTime=0
[junit4:junit4] 2> 148480 T461 C203 P52410 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 148480 T461 C203 P52410 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 148480 T461 C203 P52410 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 148481 T461 C203 P52410 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ca6b5ce0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a7b682d),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ca6b5ce0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a7b682d),segFN=segments_2,generation=2}
[junit4:junit4] 2> 148482 T461 C203 P52410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 148482 T461 C203 P52410 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 148483 T461 C203 P52410 oass.SolrIndexSearcher.<init> Opening Searcher@a647dd8f main
[junit4:junit4] 2> 148484 T460 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a647dd8f main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 148484 T461 C203 P52410 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index.20130618133630694 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index.20130618133630694;done=true>>]
[junit4:junit4] 2> 148484 T461 C203 P52410 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index.20130618133630694
[junit4:junit4] 2> 148485 T461 C203 P52410 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index.20130618133630694
[junit4:junit4] 2> 148485 T461 C203 P52410 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 148485 T461 C203 P52410 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 148485 T461 C203 P52410 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 148485 T461 C203 P52410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 148602 T461 C203 P52410 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 148651 T367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 149145 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 149146 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46951___onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 149149 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52410/_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "node_name":"127.0.0.1:52410__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 149152 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 149152 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 149152 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 149152 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 149153 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 149153 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 149653 T367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 149654 T367 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 149654 T367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 149665 T367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 149666 T469 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c9b2f687 name:ZooKeeperConnection Watcher:127.0.0.1:45859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 149667 T367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 149716 T367 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50784
[junit4:junit4] 2> 149717 T367 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1120475356
[junit4:junit4] 2> 150657 T390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 150658 T390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50784/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50784__",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 150661 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 150661 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 150661 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 150661 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 150661 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 151722 T367 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 151723 T367 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 151725 T367 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1505affa
[junit4:junit4] 2> 151731 T367 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 151732 T367 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 151733 T367 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 151734 T367 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 151735 T367 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 151736 T367 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 151736 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data;done=false>>]
[junit4:junit4] 2> 151737 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data
[junit4:junit4] 2> 151738 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/index;done=false>>]
[junit4:junit4] 2> 151739 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/control/data/index
[junit4:junit4] 2> 151740 T390 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888612199366659-127.0.0.1:50784__-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 151741 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 151741 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 151742 T425 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 151742 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 151743 T409 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 151743 T458 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 151744 T441 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 151745 T409 oasc.Overseer.start Overseer (id=89888612199366662-127.0.0.1:46951__-n_0000000001) starting
[junit4:junit4] 2> 151748 T471 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 151749 T409 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 151749 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 151749 T470 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 151751 T470 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 151752 T470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50784/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50784__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 151763 T367 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_,null}
[junit4:junit4] 2> 151815 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 151815 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 151815 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 151815 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 151817 T367 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 46951
[junit4:junit4] 2> 151817 T367 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1176669112
[junit4:junit4] 2> 153041 T389 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 153041 T389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 153042 T389 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 153319 T470 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 153320 T470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46951___onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 153423 T470 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 153430 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 153430 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 153430 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 153430 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 154822 T367 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 154823 T367 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 154824 T367 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 154826 T367 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@ab15e19
[junit4:junit4] 2> 154831 T367 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 154832 T367 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 154832 T367 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 154833 T367 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 154834 T367 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 154835 T367 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 154836 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1;done=false>>]
[junit4:junit4] 2> 154836 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1
[junit4:junit4] 2> 154837 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/index;done=false>>]
[junit4:junit4] 2> 154838 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty1/index
[junit4:junit4] 2> 154839 T367 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@f6f5f9fd
[junit4:junit4] 2> 154846 T367 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 154847 T367 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 154848 T367 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 154849 T367 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 154850 T367 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 154850 T367 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 154851 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/index;done=false>>]
[junit4:junit4] 2> 154852 T367 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection/index
[junit4:junit4] 2> 154853 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection;done=false>>]
[junit4:junit4] 2> 154854 T367 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1371591372668/onenodecollection
[junit4:junit4] 2> 154857 T470 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888612199366662-127.0.0.1:46951__-n_0000000001) am no longer a leader.
[junit4:junit4] 2> 154861 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 154861 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 154863 T425 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 154863 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 154864 T441 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 154865 T458 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 154866 T441 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 154866 T425 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 154869 T441 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 154869 T441 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 154869 T441 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 154870 T441 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60467/_/collection1/
[junit4:junit4] 2> 154870 T441 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 154870 T441 oasc.SyncStrategy.syncToMe http://127.0.0.1:60467/_/collection1/ has no replicas
[junit4:junit4] 2> 154871 T441 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60467/_/collection1/
[junit4:junit4] 2> 154871 T425 oasc.Overseer.start Overseer (id=89888612199366664-127.0.0.1:43159__-n_0000000002) starting
[junit4:junit4] 2> 154871 T441 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 154882 T473 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 154884 T472 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 154886 T367 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_,null}
[junit4:junit4] 2> 154897 T472 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 154899 T472 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 154905 T472 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46951___onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 154916 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 154916 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 154916 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 154939 T367 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 43159
[junit4:junit4] 2> 154940 T367 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=241613837
[junit4:junit4] 2> 155946 T367 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 155947 T367 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 155950 T367 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@17a44e70
[junit4:junit4] 2> 155957 T367 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 155958 T367 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 155962 T367 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 155962 T367 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 155965 T367 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 155966 T367 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 155967 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2;done=false>>]
[junit4:junit4] 2> 155968 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2
[junit4:junit4] 2> 155970 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/index;done=false>>]
[junit4:junit4] 2> 155972 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty2/index
[junit4:junit4] 2> 155976 T472 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888612199366664-127.0.0.1:43159__-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 155980 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 155983 T458 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 155985 T441 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 155987 T458 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 155987 T458 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 155988 T458 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 155988 T458 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52410/_/collection1/
[junit4:junit4] 2> 155989 T458 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52410/_ START replicas=[http://127.0.0.1:43159/_/collection1/] nUpdates=100
[junit4:junit4] 2> 155989 T458 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52410/_ DONE. We have no versions. sync failed.
[junit4:junit4] 2> 155990 T458 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 155990 T458 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 155990 T458 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52410/_/collection1/
[junit4:junit4] 2> 155991 T458 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 155992 T441 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 155995 T441 oasc.Overseer.start Overseer (id=89888612199366666-127.0.0.1:60467__-n_0000000003) starting
[junit4:junit4] 2> 156004 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 156006 T458 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 156010 T475 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 156012 T367 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_,null}
[junit4:junit4] 2> 156015 T475 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 156017 T475 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43159/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:43159__",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 156018 T476 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 156026 T475 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43159/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:43159__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 156042 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 156045 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 156063 T367 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60467
[junit4:junit4] 2> 156064 T367 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1795795558
[junit4:junit4] 2> 156163 T409 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 156163 T409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 156163 T409 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 157066 T367 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 157067 T367 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 157072 T367 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@f472a525
[junit4:junit4] 2> 157078 T367 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 157078 T367 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 157079 T367 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 157080 T367 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 157082 T367 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 157083 T367 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 157083 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index;done=false>>]
[junit4:junit4] 2> 157084 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3/index
[junit4:junit4] 2> 157085 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3;done=false>>]
[junit4:junit4] 2> 157086 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty3
[junit4:junit4] 2> 157095 T475 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888612199366666-127.0.0.1:60467__-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 157097 T441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 157097 T441 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 157097 T441 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 157098 T458 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 157100 T458 oasc.Overseer.start Overseer (id=89888612199366668-127.0.0.1:52410__-n_0000000004) starting
[junit4:junit4] 2> 157104 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 157106 T478 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 157107 T477 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 157108 T458 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 157116 T477 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 157117 T477 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60467/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:60467__",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 157118 T367 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_,null}
[junit4:junit4] 2> 157121 T477 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60467/_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "node_name":"127.0.0.1:60467__",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 157126 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 157174 T367 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 52410
[junit4:junit4] 2> 157176 T367 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2013843607
[junit4:junit4] 2> 157280 T425 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 157281 T425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 157281 T425 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 158178 T367 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 158178 T367 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 158180 T367 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@558fdb31
[junit4:junit4] 2> 158188 T367 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 158188 T367 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 158189 T367 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 158190 T367 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 158191 T367 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 158192 T367 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 158193 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4;done=false>>]
[junit4:junit4] 2> 158194 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4
[junit4:junit4] 2> 158194 T367 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index;done=false>>]
[junit4:junit4] 2> 158195 T367 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1371591372668/jetty4/index
[junit4:junit4] 2> 158197 T477 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888612199366668-127.0.0.1:52410__-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 158198 T458 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 158198 T458 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 158198 T458 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 158223 T367 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_,null}
[junit4:junit4] 2> 158280 T367 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 158282 T367 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:45859 45859
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89888612199366659-127.0.0.1:50784__-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (5)
[junit4:junit4] 1> /solr/overseer_elect/election/89888612199366668-127.0.0.1:52410__-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888612199366659-127.0.0.1:50784__-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888612199366664-127.0.0.1:43159__-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888612199366666-127.0.0.1:60467__-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888612199366662-127.0.0.1:46951__-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89888612199366659-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50784/_",
[junit4:junit4] 1> "node_name":"127.0.0.1:50784__",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888612199366662-2-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888612199366666-4-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89888612199366668-5-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89888612199366664-3-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 1> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:43159/_",
[junit4:junit4] 1> "node_name":"127.0.0.1:43159__",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89888612199366662-127.0.0.1:46951___onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46951/_",
[junit4:junit4] 1> "node_name":"127.0.0.1:46951__",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (5)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:52410__ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:43159__ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50784__ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:60467__ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46951__ (0
[...truncated too long message...]
ection/89888643455909891-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39242",
[junit4:junit4] 1> "node_name":"127.0.0.1:39242_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888643455909900-5-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888643455909902-2-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888643455909898-4-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46043",
[junit4:junit4] 1> "node_name":"127.0.0.1:46043_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:39242_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46520_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:47934_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46043_ (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39242",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:39242_",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "2":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46520",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:46520_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"},
[junit4:junit4] 1> "3":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44440",
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "node_name":"127.0.0.1:44440_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"},
[junit4:junit4] 1> "4":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:46043",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:46043_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "5":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47934",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:47934_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=34FFADCFCB0F3CD9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=th -Dtests.timezone=Brazil/East -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 103s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:46520/collection1lastClient and got 249 from http://127.0.0.1:46043/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([34FFADCFCB0F3CD9:B51923D7BC505CE5]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:780)
[junit4:junit4] 2> 709948 T1931 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 102637 T1930 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=433), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=433)), sim=DefaultSimilarity, locale=th, timezone=Brazil/East
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=43855096,total=204341248
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SolrTestCaseJ4Test, LegacyHTMLStripCharFilterTest, TestStressRecovery, TestCharFilters, OverseerCollectionProcessorTest, TestFaceting, FileBasedSpellCheckerTest, QueryParsingTest, SliceStateTest, TestSchemaResource, FieldMutatingUpdateProcessorTest, TestSolrIndexConfig, ResourceLoaderTest, UpdateRequestProcessorFactoryTest, SolrCoreTest, TestMaxScoreQueryParser, SpatialFilterTest, TermVectorComponentTest, BasicFunctionalityTest, SliceStateUpdateTest, LeaderElectionTest, PrimitiveFieldTypeTest, TestClassNameShortening, BinaryUpdateRequestHandlerTest, QueryEqualityTest, TestStressVersions, TestCloudManagedSchemaAddField, AnalysisAfterCoreReloadTest, TestDocSet, TestReversedWildcardFilterFactory, AutoCommitTest, TestPseudoReturnFields, PolyFieldTest, TestWriterPerf, TestFuzzyAnalyzedSuggestions, TestFieldCollectionResource, IndexSchemaTest, JsonLoaderTest, TestCoreContainer, TestRandomDVFaceting, ShowFileRequestHandlerTest, TestIBSimilarityFactory, ExternalFileFieldSortTest, TestSort, LoggingHandlerTest, SynonymTokenizerTest, TestSurroundQueryParser, SolrIndexSplitterTest, AlternateDirectoryTest, PeerSyncTest, TestBadConfig, TestSolrQueryParserResource, TestFastLRUCache, TestCloudManagedSchema, ZkCLITest, TestPerFieldSimilarity, TestPluginEnable, MoreLikeThisHandlerTest, TestPropInjectDefaults, SpellingQueryConverterTest, TestSolrDeletionPolicy1, TestDefaultSearchFieldResource, TestQueryUtils, RegexBoostProcessorTest, UnloadDistributedZkTest, TestSolrDeletionPolicy2, BasicDistributedZkTest, DocumentBuilderTest, TestExtendedDismaxParser, TestDocumentBuilder, SearchHandlerTest, SystemInfoHandlerTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 103.66s, 1 test, 1 failure <<< FAILURES!
[...truncated 465 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:379: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1250 tests, 1 error, 1 failure, 14 ignored (8 assumptions)
Total time: 44 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_21) - Build #
6197 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6197/
Java: 32bit/jdk1.7.0_21 -server -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:59367/tr/rp/collection1lastClient and got 258 from http://127.0.0.1:45971/tr/rp/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:59367/tr/rp/collection1lastClient and got 258 from http://127.0.0.1:45971/tr/rp/collection1
at __randomizedtesting.SeedInfo.seed([9B8F61C3FFAD5F45:1A69EFDB88F23F79]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9575 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 846675 T2346 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /tr/rp
[junit4:junit4] 2> 846680 T2346 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1371586431846
[junit4:junit4] 2> 846681 T2346 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 846681 T2347 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 846781 T2346 oasc.ZkTestServer.run start zk server on port:59491
[junit4:junit4] 2> 846782 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 846903 T2353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29387c name:ZooKeeperConnection Watcher:127.0.0.1:59491 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 846904 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 846904 T2346 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 846908 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 846909 T2355 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e5af49 name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 846910 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 846910 T2346 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 846914 T2346 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 846917 T2346 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 846919 T2346 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 846921 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 846922 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 846926 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 846926 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 846929 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 846930 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 846933 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 846934 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 846936 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 846936 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 846939 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 846940 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 846945 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 846946 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 846949 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 846950 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 846953 T2346 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 846954 T2346 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 847037 T2346 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 847041 T2346 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54843
[junit4:junit4] 2> 847042 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 847042 T2346 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 847042 T2346 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122
[junit4:junit4] 2> 847043 T2346 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/solr.xml
[junit4:junit4] 2> 847043 T2346 oasc.CoreContainer.<init> New CoreContainer 3724428
[junit4:junit4] 2> 847044 T2346 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/'
[junit4:junit4] 2> 847044 T2346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/'
[junit4:junit4] 2> 847093 T2346 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 847093 T2346 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 847093 T2346 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 847094 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 847094 T2346 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 847095 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 847095 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 847095 T2346 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 847096 T2346 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 847096 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 847099 T2346 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 847100 T2346 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59491/solr
[junit4:junit4] 2> 847100 T2346 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 847102 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 847103 T2366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21bb69 name:ZooKeeperConnection Watcher:127.0.0.1:59491 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 847103 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 847105 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 847109 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 847111 T2368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bea0d name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 847111 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 847113 T2346 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 847116 T2346 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 847119 T2346 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 847121 T2346 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54843_tr%2Frp
[junit4:junit4] 2> 847122 T2346 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54843_tr%2Frp
[junit4:junit4] 2> 847124 T2346 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 847130 T2346 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 847132 T2346 oasc.Overseer.start Overseer (id=89888288397656067-127.0.0.1:54843_tr%2Frp-n_0000000000) starting
[junit4:junit4] 2> 847140 T2346 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 847148 T2370 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 847148 T2346 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 847150 T2346 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 847152 T2346 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 847168 T2371 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 847168 T2371 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 847168 T2369 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 847175 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 847176 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54843_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54843/tr/rp"}
[junit4:junit4] 2> 847176 T2369 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 847176 T2369 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 847179 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 848171 T2371 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/collection1
[junit4:junit4] 2> 848172 T2371 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 848173 T2371 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 848173 T2371 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 848174 T2371 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/collection1/'
[junit4:junit4] 2> 848175 T2371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/collection1/lib/README' to classloader
[junit4:junit4] 2> 848175 T2371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 848225 T2371 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 848273 T2371 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 848274 T2371 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 848279 T2371 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 848702 T2371 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 848707 T2371 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 848709 T2371 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 848721 T2371 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 848725 T2371 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 848729 T2371 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 848730 T2371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 848730 T2371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 848730 T2371 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 848731 T2371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 848731 T2371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 848732 T2371 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 848742 T2371 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371586432122/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/
[junit4:junit4] 2> 848743 T2371 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5385c2
[junit4:junit4] 2> 848743 T2371 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 848743 T2371 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data
[junit4:junit4] 2> 848743 T2371 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index/
[junit4:junit4] 2> 848744 T2371 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 848744 T2371 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index
[junit4:junit4] 2> 848748 T2371 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 848748 T2371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 848750 T2371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 848751 T2371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 848751 T2371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 848752 T2371 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 848752 T2371 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 848752 T2371 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 848753 T2371 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 848753 T2371 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 848753 T2371 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 848757 T2371 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 848760 T2371 oass.SolrIndexSearcher.<init> Opening Searcher@c00f40 main
[junit4:junit4] 2> 848761 T2371 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 848761 T2371 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 848766 T2372 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c00f40 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 848767 T2371 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 848767 T2371 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54843/tr/rp collection:control_collection shard:shard1
[junit4:junit4] 2> 848768 T2371 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 848808 T2371 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 848811 T2371 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 848811 T2371 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 848811 T2371 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54843/tr/rp/collection1/
[junit4:junit4] 2> 848811 T2371 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 848811 T2371 oasc.SyncStrategy.syncToMe http://127.0.0.1:54843/tr/rp/collection1/ has no replicas
[junit4:junit4] 2> 848812 T2371 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54843/tr/rp/collection1/
[junit4:junit4] 2> 848812 T2371 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 850189 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 850200 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 850242 T2371 oasc.ZkController.register We are http://127.0.0.1:54843/tr/rp/collection1/ and leader is http://127.0.0.1:54843/tr/rp/collection1/
[junit4:junit4] 2> 850242 T2371 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54843/tr/rp
[junit4:junit4] 2> 850242 T2371 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 850242 T2371 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 850243 T2371 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 850245 T2371 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 850247 T2346 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 850248 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 850248 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 850252 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 850254 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 850256 T2375 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a3e392 name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 850257 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 850260 T2346 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 850263 T2346 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 850348 T2346 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 850350 T2346 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59367
[junit4:junit4] 2> 850351 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 850351 T2346 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 850351 T2346 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429
[junit4:junit4] 2> 850352 T2346 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/solr.xml
[junit4:junit4] 2> 850352 T2346 oasc.CoreContainer.<init> New CoreContainer 13614852
[junit4:junit4] 2> 850353 T2346 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/'
[junit4:junit4] 2> 850353 T2346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/'
[junit4:junit4] 2> 850414 T2346 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 850415 T2346 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 850415 T2346 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 850415 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 850416 T2346 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 850416 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 850416 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 850417 T2346 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 850417 T2346 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 850417 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 850420 T2346 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 850421 T2346 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59491/solr
[junit4:junit4] 2> 850421 T2346 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 850422 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 850423 T2386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90dda1 name:ZooKeeperConnection Watcher:127.0.0.1:59491 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 850424 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 850426 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 850429 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 850430 T2388 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1130a3b name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 850431 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 850434 T2346 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 851436 T2346 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59367_tr%2Frp
[junit4:junit4] 2> 851437 T2346 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59367_tr%2Frp
[junit4:junit4] 2> 851440 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 851440 T2375 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 851440 T2388 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 851441 T2368 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 851444 T2389 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 851444 T2389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 851705 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 851705 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:54843_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54843_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54843/tr/rp"}
[junit4:junit4] 2> 851708 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59367_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59367/tr/rp"}
[junit4:junit4] 2> 851708 T2369 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 851708 T2369 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 851711 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 851711 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 851711 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 852445 T2389 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/collection1
[junit4:junit4] 2> 852446 T2389 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 852446 T2389 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 852447 T2389 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 852448 T2389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/collection1/'
[junit4:junit4] 2> 852448 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/collection1/lib/README' to classloader
[junit4:junit4] 2> 852448 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 852471 T2389 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 852498 T2389 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 852499 T2389 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 852502 T2389 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 852811 T2389 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 852816 T2389 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 852818 T2389 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 852827 T2389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 852830 T2389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 852832 T2389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 852833 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 852833 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 852833 T2389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 852834 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 852834 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 852834 T2389 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 852835 T2389 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371586435429/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/
[junit4:junit4] 2> 852835 T2389 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5385c2
[junit4:junit4] 2> 852835 T2389 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 852835 T2389 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1
[junit4:junit4] 2> 852836 T2389 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index/
[junit4:junit4] 2> 852836 T2389 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 852836 T2389 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index
[junit4:junit4] 2> 852839 T2389 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 852839 T2389 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 852840 T2389 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 852841 T2389 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 852841 T2389 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 852841 T2389 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 852842 T2389 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 852842 T2389 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 852842 T2389 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 852843 T2389 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 852843 T2389 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 852845 T2389 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 852847 T2389 oass.SolrIndexSearcher.<init> Opening Searcher@1c871f1 main
[junit4:junit4] 2> 852848 T2389 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 852848 T2389 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 852851 T2390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c871f1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 852852 T2389 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 852852 T2389 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59367/tr/rp collection:collection1 shard:shard1
[junit4:junit4] 2> 852853 T2389 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 852858 T2389 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 852860 T2389 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 852860 T2389 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 852860 T2389 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 852860 T2389 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 852860 T2389 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/ has no replicas
[junit4:junit4] 2> 852860 T2389 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 852861 T2389 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 853216 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 853222 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 853222 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 853222 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 853265 T2389 oasc.ZkController.register We are http://127.0.0.1:59367/tr/rp/collection1/ and leader is http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 853266 T2389 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59367/tr/rp
[junit4:junit4] 2> 853266 T2389 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 853266 T2389 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 853266 T2389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 853268 T2389 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 853269 T2346 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 853269 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 853269 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 853333 T2346 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 853335 T2346 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44396
[junit4:junit4] 2> 853336 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 853336 T2346 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 853337 T2346 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438
[junit4:junit4] 2> 853337 T2346 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/solr.xml
[junit4:junit4] 2> 853337 T2346 oasc.CoreContainer.<init> New CoreContainer 23868326
[junit4:junit4] 2> 853338 T2346 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/'
[junit4:junit4] 2> 853338 T2346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/'
[junit4:junit4] 2> 853386 T2346 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 853387 T2346 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 853387 T2346 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 853387 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 853387 T2346 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 853388 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 853388 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 853388 T2346 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 853389 T2346 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 853389 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 853392 T2346 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 853392 T2346 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59491/solr
[junit4:junit4] 2> 853393 T2346 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 853393 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 853395 T2402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8f037c name:ZooKeeperConnection Watcher:127.0.0.1:59491 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 853395 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 853397 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 853399 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 853401 T2404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@415dc5 name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 853401 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 853404 T2346 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 854407 T2346 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44396_tr%2Frp
[junit4:junit4] 2> 854408 T2346 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44396_tr%2Frp
[junit4:junit4] 2> 854411 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 854411 T2375 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 854412 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 854412 T2388 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 854412 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 854413 T2404 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 854413 T2368 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 854417 T2405 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 854417 T2405 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 854726 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 854727 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:59367_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59367_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59367/tr/rp"}
[junit4:junit4] 2> 854730 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44396_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44396/tr/rp"}
[junit4:junit4] 2> 854730 T2369 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 854730 T2369 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 854733 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 854733 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 854733 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 854733 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 855418 T2405 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/collection1
[junit4:junit4] 2> 855418 T2405 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 855419 T2405 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 855419 T2405 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 855420 T2405 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/collection1/'
[junit4:junit4] 2> 855421 T2405 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/collection1/lib/README' to classloader
[junit4:junit4] 2> 855421 T2405 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 855443 T2405 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 855481 T2405 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 855482 T2405 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 855485 T2405 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 855785 T2405 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 855790 T2405 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 855792 T2405 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 855801 T2405 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 855804 T2405 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 855807 T2405 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 855808 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 855808 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 855808 T2405 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 855809 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 855809 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 855809 T2405 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 855810 T2405 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371586438438/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/
[junit4:junit4] 2> 855810 T2405 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5385c2
[junit4:junit4] 2> 855810 T2405 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 855811 T2405 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2
[junit4:junit4] 2> 855818 T2405 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index/
[junit4:junit4] 2> 855818 T2405 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 855819 T2405 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index
[junit4:junit4] 2> 855822 T2405 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 855822 T2405 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 855823 T2405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 855824 T2405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 855824 T2405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 855824 T2405 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 855825 T2405 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 855825 T2405 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 855825 T2405 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 855826 T2405 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 855826 T2405 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 855828 T2405 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 855830 T2405 oass.SolrIndexSearcher.<init> Opening Searcher@16e236a main
[junit4:junit4] 2> 855831 T2405 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 855831 T2405 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 855835 T2406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16e236a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 855837 T2405 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 855837 T2405 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44396/tr/rp collection:collection1 shard:shard1
[junit4:junit4] 2> 855839 T2405 oasc.ZkController.register We are http://127.0.0.1:44396/tr/rp/collection1/ and leader is http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 855840 T2405 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44396/tr/rp
[junit4:junit4] 2> 855840 T2405 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 855840 T2405 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3688 name=collection1 org.apache.solr.core.SolrCore@1f53110 url=http://127.0.0.1:44396/tr/rp/collection1 node=127.0.0.1:44396_tr%2Frp C3688_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:44396_tr%2Frp, base_url=http://127.0.0.1:44396/tr/rp}
[junit4:junit4] 2> 855840 T2407 C3688 P44396 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 855841 T2407 C3688 P44396 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 855841 T2405 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 855841 T2407 C3688 P44396 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 855842 T2407 C3688 P44396 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 855842 T2346 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 855842 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 855842 T2407 C3688 P44396 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 855843 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 855848 T2381 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:44396_tr%2Frp_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 855909 T2346 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 855911 T2346 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45971
[junit4:junit4] 2> 855912 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 855912 T2346 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 855912 T2346 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013
[junit4:junit4] 2> 855913 T2346 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/solr.xml
[junit4:junit4] 2> 855913 T2346 oasc.CoreContainer.<init> New CoreContainer 22746897
[junit4:junit4] 2> 855913 T2346 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/'
[junit4:junit4] 2> 855914 T2346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/'
[junit4:junit4] 2> 855951 T2346 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 855951 T2346 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 855952 T2346 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 855952 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 855952 T2346 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 855953 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 855953 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 855953 T2346 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 855953 T2346 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 855954 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 855956 T2346 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 855957 T2346 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59491/solr
[junit4:junit4] 2> 855957 T2346 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 855958 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 855959 T2419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1102f1c name:ZooKeeperConnection Watcher:127.0.0.1:59491 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 855960 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 855962 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 855965 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 855966 T2421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19169e1 name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 855966 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 855969 T2346 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 856237 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 856238 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44396_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44396_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44396/tr/rp"}
[junit4:junit4] 2> 856241 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856241 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856241 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856241 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856241 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856848 T2381 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:44396_tr%2Frp_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 856849 T2381 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:44396_tr%252Frp_collection1&state=recovering&nodeName=127.0.0.1:44396_tr%252Frp&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 856971 T2346 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45971_tr%2Frp
[junit4:junit4] 2> 856972 T2346 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45971_tr%2Frp
[junit4:junit4] 2> 856974 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856974 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 856974 T2388 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 856974 T2421 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 856975 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 856975 T2375 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 856975 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 856975 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 856976 T2368 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 856976 T2404 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 856981 T2422 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 856981 T2422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 857745 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 857746 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45971_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45971/tr/rp"}
[junit4:junit4] 2> 857746 T2369 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 857746 T2369 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 857749 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 857749 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 857750 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 857749 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 857749 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 857982 T2422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/collection1
[junit4:junit4] 2> 857982 T2422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 857983 T2422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 857983 T2422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 857985 T2422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/collection1/'
[junit4:junit4] 2> 857985 T2422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/collection1/lib/README' to classloader
[junit4:junit4] 2> 857986 T2422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 858016 T2422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 858058 T2422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 858059 T2422 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 858063 T2422 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 858388 T2422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 858392 T2422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 858395 T2422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 858406 T2422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 858408 T2422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 858411 T2422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 858412 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 858413 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 858413 T2422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 858414 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 858414 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 858415 T2422 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 858415 T2422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371586441013/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/
[junit4:junit4] 2> 858415 T2422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5385c2
[junit4:junit4] 2> 858415 T2422 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 858416 T2422 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3
[junit4:junit4] 2> 858416 T2422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index/
[junit4:junit4] 2> 858416 T2422 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 858416 T2422 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index
[junit4:junit4] 2> 858419 T2422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 858419 T2422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 858421 T2422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 858421 T2422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 858422 T2422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 858422 T2422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 858422 T2422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 858423 T2422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 858423 T2422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 858423 T2422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 858423 T2422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 858426 T2422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 858428 T2422 oass.SolrIndexSearcher.<init> Opening Searcher@1e99bef main
[junit4:junit4] 2> 858429 T2422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 858429 T2422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 858432 T2423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e99bef main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 858434 T2422 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 858434 T2422 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45971/tr/rp collection:collection1 shard:shard1
[junit4:junit4] 2> 858437 T2422 oasc.ZkController.register We are http://127.0.0.1:45971/tr/rp/collection1/ and leader is http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 858437 T2422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45971/tr/rp
[junit4:junit4] 2> 858438 T2422 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 858438 T2422 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3689 name=collection1 org.apache.solr.core.SolrCore@6ff8a9 url=http://127.0.0.1:45971/tr/rp/collection1 node=127.0.0.1:45971_tr%2Frp C3689_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:45971_tr%2Frp, base_url=http://127.0.0.1:45971/tr/rp}
[junit4:junit4] 2> 858438 T2424 C3689 P45971 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 858438 T2422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 858439 T2424 C3689 P45971 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 858439 T2424 C3689 P45971 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 858439 T2424 C3689 P45971 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 858440 T2346 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 858440 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 858440 T2424 C3689 P45971 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 858440 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 858444 T2384 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:45971_tr%2Frp_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 858503 T2346 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 858506 T2346 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38522
[junit4:junit4] 2> 858507 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 858507 T2346 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 858507 T2346 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609
[junit4:junit4] 2> 858508 T2346 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/solr.xml
[junit4:junit4] 2> 858508 T2346 oasc.CoreContainer.<init> New CoreContainer 18118791
[junit4:junit4] 2> 858508 T2346 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/'
[junit4:junit4] 2> 858509 T2346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/'
[junit4:junit4] 2> 858545 T2346 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 858546 T2346 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 858546 T2346 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 858546 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 858547 T2346 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 858547 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 858547 T2346 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 858548 T2346 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 858548 T2346 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 858548 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 858551 T2346 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 858551 T2346 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59491/solr
[junit4:junit4] 2> 858552 T2346 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 858553 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 858554 T2436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b65e58 name:ZooKeeperConnection Watcher:127.0.0.1:59491 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 858555 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 858556 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 858559 T2346 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 858560 T2438 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7daec3 name:ZooKeeperConnection Watcher:127.0.0.1:59491/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 858560 T2346 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 858563 T2346 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C3690 name=collection1 org.apache.solr.core.SolrCore@1f53110 url=http://127.0.0.1:44396/tr/rp/collection1 node=127.0.0.1:44396_tr%2Frp C3690_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:44396_tr%2Frp, base_url=http://127.0.0.1:44396/tr/rp}
[junit4:junit4] 2> 858849 T2407 C3690 P44396 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59367/tr/rp/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 858850 T2407 C3690 P44396 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44396/tr/rp START replicas=[http://127.0.0.1:59367/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 858850 T2407 C3690 P44396 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 858851 T2407 C3690 P44396 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 858851 T2407 C3690 P44396 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 858851 T2407 C3690 P44396 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 858851 T2407 C3690 P44396 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 858851 T2407 C3690 P44396 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59367/tr/rp/collection1/. core=collection1
[junit4:junit4] 2> 858851 T2407 C3690 P44396 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3691 name=collection1 org.apache.solr.core.SolrCore@12202eb url=http://127.0.0.1:59367/tr/rp/collection1 node=127.0.0.1:59367_tr%2Frp C3691_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59367_tr%2Frp, base_url=http://127.0.0.1:59367/tr/rp, leader=true}
[junit4:junit4] 2> 858865 T2383 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 858868 T2381 C3691 P59367 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 858870 T2381 C3691 P59367 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 858870 T2381 C3691 P59367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 858882 T2381 C3691 P59367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 858883 T2381 C3691 P59367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 858884 T2381 C3691 P59367 oass.SolrIndexSearcher.<init> Opening Searcher@181a6b2 realtime
[junit4:junit4] 2> 858884 T2381 C3691 P59367 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 858884 T2381 C3691 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> ASYNC NEW_CORE C3692 name=collection1 org.apache.solr.core.SolrCore@1f53110 url=http://127.0.0.1:44396/tr/rp/collection1 node=127.0.0.1:44396_tr%2Frp C3692_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:44396_tr%2Frp, base_url=http://127.0.0.1:44396/tr/rp}
[junit4:junit4] 2> 858885 T2407 C3692 P44396 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 858885 T2407 C3692 P44396 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 858887 T2382 C3691 P59367 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 858887 T2382 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 858888 T2407 C3692 P44396 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 858888 T2407 C3692 P44396 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 858888 T2407 C3692 P44396 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 858890 T2381 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 858890 T2407 C3692 P44396 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 858891 T2407 C3692 P44396 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index.20130618231404057
[junit4:junit4] 2> 858891 T2407 C3692 P44396 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index.20130618231404057 lockFactory=org.apache.lucene.store.NativeFSLockFactory@100fda8; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 858893 T2383 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 858894 T2407 C3692 P44396 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 858903 T2407 C3692 P44396 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 858903 T2407 C3692 P44396 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 858905 T2407 C3692 P44396 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 858905 T2407 C3692 P44396 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 858905 T2407 C3692 P44396 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 858906 T2407 C3692 P44396 oass.SolrIndexSearcher.<init> Opening Searcher@ae7482 main
[junit4:junit4] 2> 858906 T2406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ae7482 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 858907 T2407 C3692 P44396 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index.20130618231404057 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index.20130618231404057;done=true>>]
[junit4:junit4] 2> 858907 T2407 C3692 P44396 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index.20130618231404057
[junit4:junit4] 2> 858907 T2407 C3692 P44396 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index.20130618231404057
[junit4:junit4] 2> 858908 T2407 C3692 P44396 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 858908 T2407 C3692 P44396 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 858908 T2407 C3692 P44396 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 858908 T2407 C3692 P44396 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 858910 T2407 C3692 P44396 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 859253 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 859254 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45971_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45971_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45971/tr/rp"}
[junit4:junit4] 2> 859257 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44396_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44396_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44396/tr/rp"}
[junit4:junit4] 2> 859276 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859276 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859277 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859277 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859277 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859277 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859444 T2384 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:45971_tr%2Frp_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 859445 T2384 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:45971_tr%252Frp_collection1&state=recovering&nodeName=127.0.0.1:45971_tr%252Frp&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 859565 T2346 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38522_tr%2Frp
[junit4:junit4] 2> 859567 T2346 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38522_tr%2Frp
[junit4:junit4] 2> 859569 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859569 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 859570 T2421 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 859570 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 859570 T2388 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 859570 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 859570 T2438 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 859571 T2375 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 859571 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 859571 T2404 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 859572 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 859572 T2368 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 859577 T2441 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 859577 T2441 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 860782 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 860783 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38522_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38522/tr/rp"}
[junit4:junit4] 2> 860783 T2369 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 860783 T2369 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 860786 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 860786 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 860786 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 860786 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 860786 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 860787 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C3693 name=collection1 org.apache.solr.core.SolrCore@6ff8a9 url=http://127.0.0.1:45971/tr/rp/collection1 node=127.0.0.1:45971_tr%2Frp C3693_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:45971_tr%2Frp, base_url=http://127.0.0.1:45971/tr/rp}
[junit4:junit4] 2> 861445 T2424 C3693 P45971 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59367/tr/rp/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 861446 T2424 C3693 P45971 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45971/tr/rp START replicas=[http://127.0.0.1:59367/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 861446 T2424 C3693 P45971 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 861446 T2424 C3693 P45971 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 861447 T2424 C3693 P45971 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 861447 T2424 C3693 P45971 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 861447 T2424 C3693 P45971 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 861447 T2424 C3693 P45971 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59367/tr/rp/collection1/. core=collection1
[junit4:junit4] 2> 861447 T2424 C3693 P45971 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 861448 T2381 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 861450 T2383 C3691 P59367 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861463 T2383 C3691 P59367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 861463 T2383 C3691 P59367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 861463 T2383 C3691 P59367 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861464 T2383 C3691 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 861464 T2424 C3693 P45971 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 861464 T2424 C3693 P45971 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 861466 T2384 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 861467 T2424 C3693 P45971 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 861467 T2424 C3693 P45971 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 861467 T2424 C3693 P45971 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 861468 T2381 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 861469 T2424 C3693 P45971 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 861470 T2424 C3693 P45971 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index.20130618231406635
[junit4:junit4] 2> 861470 T2424 C3693 P45971 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index.20130618231406635 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9d4bcd; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 861472 T2383 C3691 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 861473 T2424 C3693 P45971 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 861475 T2424 C3693 P45971 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 861475 T2424 C3693 P45971 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 861477 T2424 C3693 P45971 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 861477 T2424 C3693 P45971 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 861477 T2424 C3693 P45971 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 861478 T2424 C3693 P45971 oass.SolrIndexSearcher.<init> Opening Searcher@5d1de7 main
[junit4:junit4] 2> 861478 T2423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d1de7 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 861479 T2424 C3693 P45971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index.20130618231406635 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index.20130618231406635;done=true>>]
[junit4:junit4] 2> 861479 T2424 C3693 P45971 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index.20130618231406635
[junit4:junit4] 2> 861479 T2424 C3693 P45971 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index.20130618231406635
[junit4:junit4] 2> 861479 T2424 C3693 P45971 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 861479 T2424 C3693 P45971 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 861480 T2424 C3693 P45971 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 861480 T2424 C3693 P45971 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 861481 T2424 C3693 P45971 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 861578 T2441 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/collection1
[junit4:junit4] 2> 861579 T2441 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 861579 T2441 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 861580 T2441 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 861581 T2441 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/collection1/'
[junit4:junit4] 2> 861581 T2441 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/collection1/lib/README' to classloader
[junit4:junit4] 2> 861581 T2441 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 861606 T2441 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 861641 T2441 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 861643 T2441 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 861646 T2441 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 861942 T2441 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 861946 T2441 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 861948 T2441 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 861958 T2441 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 861960 T2441 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 861963 T2441 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 861963 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 861964 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 861964 T2441 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 861965 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 861965 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 861965 T2441 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 861965 T2441 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371586443609/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/
[junit4:junit4] 2> 861965 T2441 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5385c2
[junit4:junit4] 2> 861966 T2441 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 861966 T2441 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4
[junit4:junit4] 2> 861966 T2441 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index/
[junit4:junit4] 2> 861966 T2441 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 861967 T2441 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index
[junit4:junit4] 2> 861984 T2441 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 861984 T2441 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 861986 T2441 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 861986 T2441 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 861987 T2441 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 861987 T2441 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 861987 T2441 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 861988 T2441 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 861989 T2441 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 861989 T2441 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 861989 T2441 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 861991 T2441 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 861994 T2441 oass.SolrIndexSearcher.<init> Opening Searcher@1c514df main
[junit4:junit4] 2> 861994 T2441 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 861995 T2441 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 861999 T2444 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c514df main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 862001 T2441 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 862001 T2441 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38522/tr/rp collection:collection1 shard:shard1
[junit4:junit4] 2> 862003 T2441 oasc.ZkController.register We are http://127.0.0.1:38522/tr/rp/collection1/ and leader is http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 862004 T2441 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38522/tr/rp
[junit4:junit4] 2> 862004 T2441 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 862004 T2441 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3694 name=collection1 org.apache.solr.core.SolrCore@c8945c url=http://127.0.0.1:38522/tr/rp/collection1 node=127.0.0.1:38522_tr%2Frp C3694_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:38522_tr%2Frp, base_url=http://127.0.0.1:38522/tr/rp}
[junit4:junit4] 2> 862005 T2445 C3694 P38522 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 862005 T2445 C3694 P38522 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 862006 T2441 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 862006 T2445 C3694 P38522 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 862006 T2445 C3694 P38522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 862007 T2346 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 862007 T2346 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 862007 T2445 C3694 P38522 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 862008 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 862020 T2381 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:38522_tr%2Frp_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 862020 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 862021 T2346 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 862022 T2346 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 862022 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 862291 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 862292 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45971_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45971_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45971/tr/rp"}
[junit4:junit4] 2> 862294 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38522_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38522_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38522/tr/rp"}
[junit4:junit4] 2> 862296 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 862297 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 862297 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 862297 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 862297 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 862297 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 863020 T2381 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:38522_tr%2Frp_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 863021 T2381 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:38522_tr%252Frp_collection1&state=recovering&nodeName=127.0.0.1:38522_tr%252Frp&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 863024 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 864025 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C3695 name=collection1 org.apache.solr.core.SolrCore@c8945c url=http://127.0.0.1:38522/tr/rp/collection1 node=127.0.0.1:38522_tr%2Frp C3695_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:38522_tr%2Frp, base_url=http://127.0.0.1:38522/tr/rp}
[junit4:junit4] 2> 865021 T2445 C3695 P38522 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59367/tr/rp/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 865022 T2445 C3695 P38522 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38522/tr/rp START replicas=[http://127.0.0.1:59367/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 865022 T2445 C3695 P38522 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 865022 T2445 C3695 P38522 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 865022 T2445 C3695 P38522 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 865022 T2445 C3695 P38522 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 865022 T2445 C3695 P38522 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 865023 T2445 C3695 P38522 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59367/tr/rp/collection1/. core=collection1
[junit4:junit4] 2> 865023 T2445 C3695 P38522 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3696 name=collection1 org.apache.solr.core.SolrCore@12202eb url=http://127.0.0.1:59367/tr/rp/collection1 node=127.0.0.1:59367_tr%2Frp C3696_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59367_tr%2Frp, base_url=http://127.0.0.1:59367/tr/rp, leader=true}
[junit4:junit4] 2> 865024 T2381 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 865025 T2382 C3696 P59367 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 865027 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 865033 T2382 C3696 P59367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 865033 T2382 C3696 P59367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 865033 T2382 C3696 P59367 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 865034 T2382 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 865034 T2445 C3695 P38522 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 865034 T2445 C3695 P38522 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 865036 T2384 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 865037 T2445 C3695 P38522 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 865037 T2445 C3695 P38522 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 865037 T2445 C3695 P38522 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 865039 T2383 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 865039 T2445 C3695 P38522 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 865040 T2445 C3695 P38522 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index.20130618231410205
[junit4:junit4] 2> 865040 T2445 C3695 P38522 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index.20130618231410205 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d8751; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 865042 T2382 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 865043 T2445 C3695 P38522 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 865045 T2445 C3695 P38522 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 865046 T2445 C3695 P38522 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 865047 T2445 C3695 P38522 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 865047 T2445 C3695 P38522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 865048 T2445 C3695 P38522 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 865048 T2445 C3695 P38522 oass.SolrIndexSearcher.<init> Opening Searcher@14a597d main
[junit4:junit4] 2> 865049 T2444 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14a597d main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 865049 T2445 C3695 P38522 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index.20130618231410205 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index.20130618231410205;done=true>>]
[junit4:junit4] 2> 865049 T2445 C3695 P38522 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index.20130618231410205
[junit4:junit4] 2> 865049 T2445 C3695 P38522 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index.20130618231410205
[junit4:junit4] 2> 865050 T2445 C3695 P38522 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 865050 T2445 C3695 P38522 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 865050 T2445 C3695 P38522 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 865050 T2445 C3695 P38522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 865051 T2445 C3695 P38522 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 865303 T2369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 865304 T2369 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38522_tr%2Frp_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38522_tr%2Frp",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38522/tr/rp"}
[junit4:junit4] 2> 865323 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 865323 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 865323 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 865323 T2375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 865323 T2368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 865323 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 866028 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 866029 T2346 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C3697 name=collection1 org.apache.solr.core.SolrCore@1451d url=http://127.0.0.1:54843/tr/rp/collection1 node=127.0.0.1:54843_tr%2Frp C3697_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:54843_tr%2Frp, base_url=http://127.0.0.1:54843/tr/rp, leader=true}
[junit4:junit4] 2> 866038 T2362 C3697 P54843 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 866041 T2362 C3697 P54843 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 866042 T2362 C3697 P54843 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 866053 T2362 C3697 P54843 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 866053 T2362 C3697 P54843 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 866054 T2362 C3697 P54843 oass.SolrIndexSearcher.<init> Opening Searcher@1bd45c5 main
[junit4:junit4] 2> 866054 T2362 C3697 P54843 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 866055 T2372 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bd45c5 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 866055 T2362 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 17
[junit4:junit4] 2> ASYNC NEW_CORE C3698 name=collection1 org.apache.solr.core.SolrCore@6ff8a9 url=http://127.0.0.1:45971/tr/rp/collection1 node=127.0.0.1:45971_tr%2Frp C3698_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45971_tr%2Frp, base_url=http://127.0.0.1:45971/tr/rp}
[junit4:junit4] 2> 866061 T2417 C3698 P45971 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:59367/tr/rp/collection1/, StdNode: http://127.0.0.1:44396/tr/rp/collection1/, StdNode: http://127.0.0.1:45971/tr/rp/collection1/, StdNode: http://127.0.0.1:38522/tr/rp/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 866064 T2415 C3698 P45971 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 866064 T2383 C3696 P59367 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3699 name=collection1 org.apache.solr.core.SolrCore@1f53110 url=http://127.0.0.1:44396/tr/rp/collection1 node=127.0.0.1:44396_tr%2Frp C3699_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:44396_tr%2Frp, base_url=http://127.0.0.1:44396/tr/rp}
[junit4:junit4] 2> 866068 T2400 C3699 P44396 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C3695_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38522_tr%2Frp, base_url=http://127.0.0.1:38522/tr/rp}
[junit4:junit4] 2> 866072 T2432 C3695 P38522 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 866082 T2415 C3698 P45971 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 866083 T2415 C3698 P45971 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 866084 T2415 C3698 P45971 oass.SolrIndexSearcher.<init> Opening Searcher@1132201 main
[junit4:junit4] 2> 866084 T2415 C3698 P45971 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 866085 T2423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1132201 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 866085 T2415 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> 866103 T2400 C3699 P44396 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 866103 T2383 C3696 P59367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 866103 T2383 C3696 P59367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 866103 T2400 C3699 P44396 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 866104 T2383 C3696 P59367 oass.SolrIndexSearcher.<init> Opening Searcher@9572cd main
[junit4:junit4] 2> 866105 T2400 C3699 P44396 oass.SolrIndexSearcher.<init> Opening Searcher@1af71bf main
[junit4:junit4] 2> 866105 T2383 C3696 P59367 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 866105 T2400 C3699 P44396 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 866106 T2406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1af71bf main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 866105 T2390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9572cd main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 866106 T2400 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 38
[junit4:junit4] 2> 866106 T2383 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 42
[junit4:junit4] 2> 866123 T2432 C3695 P38522 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 866123 T2432 C3695 P38522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 866124 T2432 C3695 P38522 oass.SolrIndexSearcher.<init> Opening Searcher@187fbc2 main
[junit4:junit4] 2> 866124 T2432 C3695 P38522 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 866125 T2444 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@187fbc2 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 866125 T2432 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 53
[junit4:junit4] 2> 866126 T2417 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 64
[junit4:junit4] 2> 866126 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 866128 T2384 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 866129 T2398 C3699 P44396 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 866130 T2416 C3698 P45971 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 866132 T2434 C3695 P38522 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 868136 T2361 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438212636855500800)} 0 2
[junit4:junit4] 2> 868142 T2415 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={update.distrib=FROMLEADER&_version_=-1438212636861792256&update.from=http://127.0.0.1:59367/tr/rp/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438212636861792256)} 0 0
[junit4:junit4] 2> 868143 T2397 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={update.distrib=FROMLEADER&_version_=-1438212636861792256&update.from=http://127.0.0.1:59367/tr/rp/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438212636861792256)} 0 0
[junit4:junit4] 2> 868143 T2432 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={update.distrib=FROMLEADER&_version_=-1438212636861792256&update.from=http://127.0.0.1:59367/tr/rp/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438212636861792256)} 0 0
[junit4:junit4] 2> 868144 T2381 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438212636861792256)} 0 4
[junit4:junit4] 2> 868144 T2416 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 6
[junit4:junit4] 2> 868147 T2363 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {add=[0 (1438212636868083712)]} 0 1
[junit4:junit4] 2> 868153 T2417 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438212636872278016)]} 0 1
[junit4:junit4] 2> 868153 T2398 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438212636872278016)]} 0 1
[junit4:junit4] 2> 868153 T2431 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438212636872278016)]} 0 1
[junit4:junit4] 2> 868154 T2382 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:38522/tr/rp/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438212636872278016)]} 0 4
[junit4:junit4] 2> 868154 T2434 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {add=[0]} 0 6
[junit4:junit4] 2> 868157 T2364 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {add=[1 (1438212636879618048)]} 0 1
[junit4:junit4] 2> 868163 T2415 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438212636883812352)]} 0 0
[junit4:junit4] 2> 868163 T2399 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438212636883812352)]} 0 0
[junit4:junit4] 2> 868163 T2433 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438212636883812352)]} 0 0
[junit4:junit4] 2> 868164 T2383 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:45971/tr/rp/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438212636883812352)]} 0 3
[junit4:junit4] 2> 868165 T2414 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {add=[1]} 0 6
[junit4:junit4] 2> 868167 T2362 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {add=[2 (1438212636890103808)]} 0 0
[junit4:junit4] 2> 868170 T2384 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44396/tr/rp/collection1/]
[junit4:junit4] 2> 868170 T2384 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44396/tr/rp/collection1/ against:[http://127.0.0.1:44396/tr/rp/collection1/] result:true
[junit4:junit4] 2> 868170 T2384 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45971/tr/rp/collection1/ against:[http://127.0.0.1:44396/tr/rp/collection1/] result:false
[junit4:junit4] 2> 868171 T2384 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:38522/tr/rp/collection1/ against:[http://127.0.0.1:44396/tr/rp/collection1/] result:false
[junit4:junit4] 2> 868173 T2416 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[2 (1438212636894298112)]} 0 0
[junit4:junit4] 2> 868173 T2432 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[2 (1438212636894298112)]} 0 0
[junit4:junit4] 2> 868174 T2384 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:44396/tr/rp/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[2 (1438212636894298112)]} 0 4
[junit4:junit4] 2> 868175 T2400 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[2]} 0 7
[junit4:junit4] 2> 868177 T2361 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={wt=javabin&version=2} {add=[3 (1438212636900589568)]} 0 0
[junit4:junit4] 2> 868181 T2381 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44396/tr/rp/collection1/]
[junit4:junit4] 2> 868181 T2381 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44396/tr/rp/collection1/ against:[http://127.0.0.1:44396/tr/rp/collection1/] result:true
[junit4:junit4] 2> 868181 T2381 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45971/tr/rp/collection1/ against:[http://127.0.0.1:44396/tr/rp/collection1/] result:false
[junit4:junit4] 2> 868182 T2381 C3696 P59367 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:38522/tr/rp/collection1/ against:[http://127.0.0.1:44396/tr/rp/collection1/] result:false
[junit4:junit4] 2> 868184 T2417 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[3 (1438212636905832448)]} 0 0
[junit4:junit4] 2> 868184 T2434 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:59367/tr/rp/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[3 (1438212636905832448)]} 0 0
[junit4:junit4] 2> 868185 T2381 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={distrib.from=http://127.0.0.1:38522/tr/rp/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&wt=javabin&version=2} {add=[3 (1438212636905832448)]} 0 4
[junit4:junit4] 2> 868185 T2431 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={test.distrib.skip.servers=http://127.0.0.1:44396/tr/rp/collection1/&test.distrib.skip.servers=http://127.0.0.1:45971/tr/rp/collection1/&wt=javabin&version=2} {add=[3]} 0 6
[junit4:junit4] 2> 868187 T2363 C3697 P54843 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868203 T2363 C3697 P54843 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 868203 T2363 C3697 P54843 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 868205 T2363 C3697 P54843 oass.SolrIndexSearcher.<init> Opening Searcher@1e252b7 main
[junit4:junit4] 2> 868205 T2363 C3697 P54843 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 868206 T2372 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e252b7 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 868206 T2363 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 19
[junit4:junit4] 2> 868207 T2382 C3696 P59367 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:59367/tr/rp/collection1/, StdNode: http://127.0.0.1:44396/tr/rp/collection1/, StdNode: http://127.0.0.1:45971/tr/rp/collection1/, StdNode: http://127.0.0.1:38522/tr/rp/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 868209 T2383 C3696 P59367 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868209 T2415 C3698 P45971 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868209 T2433 C3695 P38522 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868209 T2397 C3699 P44396 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868243 T2433 C3695 P38522 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@eac409; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 868243 T2383 C3696 P59367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 868243 T2433 C3695 P38522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 868244 T2383 C3696 P59367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 868245 T2433 C3695 P38522 oass.SolrIndexSearcher.<init> Opening Searcher@1d514bf main
[junit4:junit4] 2> 868245 T2383 C3696 P59367 oass.SolrIndexSearcher.<init> Opening Searcher@e717b main
[junit4:junit4] 2> 868245 T2433 C3695 P38522 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 868246 T2383 C3696 P59367 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 868246 T2444 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d514bf main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 868246 T2390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e717b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 868247 T2433 C3695 P38522 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 38
[junit4:junit4] 2> 868247 T2383 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 38
[junit4:junit4] 2> 868263 T2415 C3698 P45971 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1818f66; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 868263 T2415 C3698 P45971 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 868263 T2397 C3699 P44396 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1faaa88; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 868264 T2397 C3699 P44396 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 868264 T2415 C3698 P45971 oass.SolrIndexSearcher.<init> Opening Searcher@1450ce main
[junit4:junit4] 2> 868265 T2415 C3698 P45971 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 868265 T2397 C3699 P44396 oass.SolrIndexSearcher.<init> Opening Searcher@1a1bd4b main
[junit4:junit4] 2> 868265 T2397 C3699 P44396 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 868265 T2423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1450ce main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 868266 T2406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a1bd4b main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 868266 T2415 C3698 P45971 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 57
[junit4:junit4] 2> 868266 T2397 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 57
[junit4:junit4] 2> 868267 T2382 C3696 P59367 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 60
[junit4:junit4] 2> 868267 T2346 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 868268 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 868269 T2346 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 868271 T2384 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 868273 T2398 C3699 P44396 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 868275 T2414 C3698 P45971 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 868277 T2432 C3695 P38522 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 868277 T2346 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 868281 T2416 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 868281 T2416 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 868284 T2381 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 868284 T2381 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 868284 T2381 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp START replicas=[http://127.0.0.1:44396/tr/rp/collection1/, http://127.0.0.1:45971/tr/rp/collection1/, http://127.0.0.1:38522/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 868285 T2399 C3699 P44396 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 868286 T2415 C3698 P45971 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 868286 T2381 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp Received 3 versions from 127.0.0.1:44396/tr/rp/collection1/
[junit4:junit4] 2> 868286 T2431 C3695 P38522 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 868286 T2381 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp Our versions are newer. ourLowThreshold=1438212636861792256 otherHigh=1438212636883812352
[junit4:junit4] 2> 868287 T2381 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp Received 5 versions from 127.0.0.1:45971/tr/rp/collection1/
[junit4:junit4] 2> 868287 T2381 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp Our versions are newer. ourLowThreshold=1438212636861792256 otherHigh=1438212636894298112
[junit4:junit4] 2> 868287 T2381 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp Received 5 versions from 127.0.0.1:38522/tr/rp/collection1/
[junit4:junit4] 2> 868287 T2381 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp Our versions are newer. ourLowThreshold=1438212636861792256 otherHigh=1438212636894298112
[junit4:junit4] 2> 868288 T2381 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59367/tr/rp DONE. sync succeeded
[junit4:junit4] 2> 868288 T2381 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 868288 T2381 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/: try and ask http://127.0.0.1:44396/tr/rp/collection1/ to sync
[junit4:junit4] 2> 868288 T2381 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/: try and ask http://127.0.0.1:45971/tr/rp/collection1/ to sync
[junit4:junit4] 2> 868289 T2381 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/: try and ask http://127.0.0.1:38522/tr/rp/collection1/ to sync
[junit4:junit4] 2> 868289 T2400 C3699 P44396 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44396/tr/rp START replicas=[http://127.0.0.1:59367/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 868290 T2414 C3698 P45971 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45971/tr/rp START replicas=[http://127.0.0.1:59367/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 868290 T2432 C3695 P38522 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38522/tr/rp START replicas=[http://127.0.0.1:59367/tr/rp/collection1/] nUpdates=100
[junit4:junit4] 2> 868291 T2383 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 868291 T2384 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 868291 T2382 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 868292 T2432 C3695 P38522 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38522/tr/rp Received 5 versions from 127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 868291 T2414 C3698 P45971 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45971/tr/rp Received 5 versions from 127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 868292 T2432 C3695 P38522 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38522/tr/rp Our versions are newer. ourLowThreshold=1438212636861792256 otherHigh=1438212636894298112
[junit4:junit4] 2> 868293 T2432 C3695 P38522 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38522/tr/rp DONE. sync succeeded
[junit4:junit4] 2> 868292 T2400 C3699 P44396 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44396/tr/rp Received 5 versions from 127.0.0.1:59367/tr/rp/collection1/
[junit4:junit4] 2> 868293 T2432 C3695 P38522 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={sync=http://127.0.0.1:59367/tr/rp/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 868292 T2414 C3698 P45971 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45971/tr/rp Our versions are newer. ourLowThreshold=1438212636861792256 otherHigh=1438212636894298112
[junit4:junit4] 2> 868294 T2381 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/: sync completed with http://127.0.0.1:38522/tr/rp/collection1/
[junit4:junit4] 2> 868293 T2400 C3699 P44396 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44396/tr/rp Requesting updates from 127.0.0.1:59367/tr/rp/collection1/n=2 versions=[1438212636905832448, 1438212636894298112]
[junit4:junit4] 2> 868294 T2414 C3698 P45971 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45971/tr/rp DONE. sync succeeded
[junit4:junit4] 2> 868294 T2414 C3698 P45971 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={sync=http://127.0.0.1:59367/tr/rp/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4
[junit4:junit4] 2> 868295 T2381 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/: sync completed with http://127.0.0.1:45971/tr/rp/collection1/
[junit4:junit4] 2> 868296 T2383 C3696 P59367 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={distrib=false&getUpdates=1438212636905832448,1438212636894298112&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 868298 T2400 C3699 P44396 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438212636894298112), 3 (1438212636905832448)]} 0 2
[junit4:junit4] 2> 868299 T2400 C3699 P44396 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44396/tr/rp DONE. sync succeeded
[junit4:junit4] 2> 868299 T2400 C3699 P44396 oasc.SolrCore.execute [collection1] webapp=/tr/rp path=/get params={sync=http://127.0.0.1:59367/tr/rp/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=10
[junit4:junit4] 2> 868299 T2381 oasc.SyncStrategy.syncToMe http://127.0.0.1:59367/tr/rp/collection1/: sync completed with http://127.0.0.1:44396/tr/rp/collection1/
[junit4:junit4] 2> 868300 T2381 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=16
[junit4:junit4] 2> 868300 T2416 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=19
[junit4:junit4] 2> 868300 T2346 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 868301 T2346 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 868301 T2346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 868302 T2346 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 868304 T2364 C3697 P54843 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868343 T2364 C3697 P54843 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@10c1068; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 868343 T2364 C3697 P54843 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 868344 T2364 C3697 P54843 oass.SolrIndexSearcher.<init> Opening Searcher@85762c main
[junit4:junit4] 2> 868344 T2364 C3697 P54843 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 868345 T2372 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@85762c main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 868345 T2364 C3697 P54843 oasup.LogUpdateProcessor.finish [collection1] webapp=/tr/rp path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 41
[junit4:junit4] 2> 868347 T2397 C3699 P44396 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:59367/tr/rp/collection1/, StdNode: http://127.0.0.1:44396/tr/rp/collection1/, StdNode: http://127.0.0.1:45971/tr/rp/collection1/, StdNode: http://127.0.0.1:38522/tr/rp/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 868349 T2384 C3696 P59367 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868350 T2417 C3698 P45971 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868350 T2398 C3699 P44396 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868351 T2431 C3695 P38522 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 868373 T2384 C3696 P59367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e64d83; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371586431846/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e6
[...truncated too long message...]
junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:38522_tr%2Frp_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:38522_tr%2Frp",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:38522/tr/rp"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:54843_tr%2Frp_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:54843_tr%2Frp",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:54843/tr/rp",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:59367_tr%2Frp (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:38522_tr%2Frp (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:54843_tr%2Frp (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:45971_tr%2Frp (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888288397656076-127.0.0.1:38522_tr%2Frp_collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888288397656074-127.0.0.1:45971_tr%2Frp_collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89888288397656078-127.0.0.1:59367_tr%2Frp_collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:45971_tr%2Frp",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45971/tr/rp"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89888288397656067-127.0.0.1:54843_tr%2Frp_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:54843_tr%2Frp",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:54843/tr/rp"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89888288397656067-127.0.0.1:54843_tr%2Frp-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888288397656076-127.0.0.1:38522_tr%2Frp-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888288397656074-127.0.0.1:45971_tr%2Frp-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89888288397656078-127.0.0.1:59367_tr%2Frp-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89888288397656067-127.0.0.1:54843_tr%2Frp-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=9B8F61C3FFAD5F45 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_MX -Dtests.timezone=Europe/Vilnius -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 102s J1 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:59367/tr/rp/collection1lastClient and got 258 from http://127.0.0.1:45971/tr/rp/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([9B8F61C3FFAD5F45:1A69EFDB88F23F79]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 948246 T2346 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 101574 T2345 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_MX, timezone=Europe/Vilnius
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_21 (32-bit)/cpus=8,threads=2,free=55442624,total=194572288
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SignatureUpdateProcessorFactoryTest, TestDefaultSearchFieldResource, BasicDistributedZk2Test, SampleTest, TestDynamicFieldResource, ChaosMonkeySafeLeaderTest, TestSolr4Spatial, TestBM25SimilarityFactory, TestFiltering, UpdateParamsTest, DateMathParserTest, TestBinaryField, TestFoldingMultitermQuery, TestPropInject, SolrIndexConfigTest, TestSolrXMLSerializer, JSONWriterTest, SortByFunctionTest, TestMaxScoreQueryParser, TestCodecSupport, XsltUpdateRequestHandlerTest, TestWordDelimiterFilterFactory, StatsComponentTest, DefaultValueUpdateProcessorTest, PingRequestHandlerTest, TestJmxIntegration, TestStressReorder, FullSolrCloudDistribCmdsTest, NoCacheHeaderTest, SpellPossibilityIteratorTest, TestSchemaNameResource, TestOmitPositions, SimplePostToolTest, TestSchemaVersionResource, TestAnalyzedSuggestions, DebugComponentTest, SolrPluginUtilsTest, OpenExchangeRatesOrgProviderTest, TestCollationField, TestLMJelinekMercerSimilarityFactory, TestLazyCores, TestSolrXml, DirectSolrSpellCheckerTest, IndexBasedSpellCheckerTest, TestSolrQueryParserResource, IndexSchemaRuntimeFieldTest, NotRequiredUniqueKeyTest, SuggesterTSTTest, CopyFieldTest, LeaderElectionIntegrationTest, TestUtils, SuggesterWFSTTest, FieldAnalysisRequestHandlerTest, TestRecovery, TestRandomFaceting, TestMultiCoreConfBootstrap, TestShardHandlerFactory, TestFieldResource, TestZkChroot, TestQueryTypes, TestHashPartitioner, StandardRequestHandlerTest, TestDistributedGrouping, TestElisionMultitermQuery, TestStressLucene, HardAutoCommitTest, CoreContainerCoreInitFailuresTest, TestLuceneMatchVersion, TestAtomicUpdateErrorCases, TestPartialUpdateDeduplication, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J1 in 102.36s, 1 test, 1 failure <<< FAILURES!
[...truncated 456 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1250 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 42 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_21 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6196 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6196/
Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
ERROR: SolrIndexSearcher opens=41 closes=40
Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=41 closes=40
at __randomizedtesting.SeedInfo.seed([650D64FB840D96FC]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:254)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:102)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<642> but was:<34>
Stack Trace:
java.lang.AssertionError: expected:<642> but was:<34>
at __randomizedtesting.SeedInfo.seed([650D64FB840D96FC:E4EBEAE3F352F6C0]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9460 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 436463 T1548 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 436468 T1548 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371582246870
[junit4:junit4] 2> 436469 T1548 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 436469 T1549 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 436569 T1548 oasc.ZkTestServer.run start zk server on port:42837
[junit4:junit4] 2> 436570 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 436766 T1555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53881802 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 436766 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 436767 T1548 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 436770 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 436772 T1557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@562fabec name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 436772 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 436772 T1548 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 436776 T1548 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 436780 T1548 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 436782 T1548 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 436785 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 436785 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 436789 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 436789 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 436792 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 436793 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 436796 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 436797 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 436799 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 436799 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 436802 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 436802 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 436805 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 436806 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 436808 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 436809 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 436811 T1548 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 436811 T1548 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 436877 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 436899 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58609
[junit4:junit4] 2> 436900 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 436900 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 436900 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215
[junit4:junit4] 2> 436901 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/solr.xml
[junit4:junit4] 2> 436901 T1548 oasc.CoreContainer.<init> New CoreContainer 1433686276
[junit4:junit4] 2> 436901 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/'
[junit4:junit4] 2> 436902 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/'
[junit4:junit4] 2> 436943 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 436944 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 436944 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 436945 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 436945 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 436945 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 436946 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 436946 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 436946 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 436947 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 436950 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 436950 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 436950 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 436951 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 436953 T1568 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1059a522 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 436953 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 436956 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 436959 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 436960 T1570 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27dc6442 name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 436960 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 436962 T1548 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 436964 T1548 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 436967 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 436968 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58609_
[junit4:junit4] 2> 436969 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58609_
[junit4:junit4] 2> 436972 T1548 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 436976 T1548 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 436978 T1548 oasc.Overseer.start Overseer (id=89888014131068931-127.0.0.1:58609_-n_0000000000) starting
[junit4:junit4] 2> 436981 T1548 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 436984 T1572 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 436984 T1548 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 436986 T1548 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 436988 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 436990 T1571 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 436992 T1573 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 436993 T1573 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 438493 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 438494 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58609_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58609"}
[junit4:junit4] 2> 438494 T1571 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 438495 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 438499 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 438994 T1573 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/collection1
[junit4:junit4] 2> 438994 T1573 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 438995 T1573 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 438996 T1573 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 438997 T1573 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/collection1/'
[junit4:junit4] 2> 438998 T1573 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/collection1/lib/README' to classloader
[junit4:junit4] 2> 438998 T1573 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 439022 T1573 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 439050 T1573 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 439052 T1573 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 439055 T1573 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 439377 T1573 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 439382 T1573 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 439385 T1573 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 439396 T1573 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 439399 T1573 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 439402 T1573 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 439403 T1573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 439403 T1573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 439403 T1573 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 439404 T1573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 439404 T1573 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 439404 T1573 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 439405 T1573 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371582247215/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/control/data/
[junit4:junit4] 2> 439405 T1573 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 439405 T1573 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 439406 T1573 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/control/data
[junit4:junit4] 2> 439406 T1573 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/control/data/index/
[junit4:junit4] 2> 439406 T1573 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 439406 T1573 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/control/data/index
[junit4:junit4] 2> 439456 T1573 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 439457 T1573 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 439458 T1573 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 439459 T1573 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 439459 T1573 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 439460 T1573 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 439460 T1573 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 439460 T1573 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 439460 T1573 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 439461 T1573 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 439461 T1573 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 439463 T1573 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 439466 T1573 oass.SolrIndexSearcher.<init> Opening Searcher@55500e4 main
[junit4:junit4] 2> 439467 T1573 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 439467 T1573 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 439470 T1574 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55500e4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 439471 T1573 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 439471 T1573 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58609 collection:control_collection shard:shard1
[junit4:junit4] 2> 439472 T1573 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 439478 T1573 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 439480 T1573 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 439480 T1573 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 439480 T1573 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58609/collection1/
[junit4:junit4] 2> 439481 T1573 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 439481 T1573 oasc.SyncStrategy.syncToMe http://127.0.0.1:58609/collection1/ has no replicas
[junit4:junit4] 2> 439481 T1573 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58609/collection1/
[junit4:junit4] 2> 439482 T1573 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 440003 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 440010 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 440038 T1573 oasc.ZkController.register We are http://127.0.0.1:58609/collection1/ and leader is http://127.0.0.1:58609/collection1/
[junit4:junit4] 2> 440038 T1573 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58609
[junit4:junit4] 2> 440039 T1573 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 440039 T1573 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 440039 T1573 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 440041 T1573 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 440042 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 440043 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 440043 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 440046 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 440047 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 440048 T1577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23cec9f5 name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 440048 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 440050 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 440052 T1548 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 440124 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 440125 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59804
[junit4:junit4] 2> 440126 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 440126 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 440127 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453
[junit4:junit4] 2> 440127 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/solr.xml
[junit4:junit4] 2> 440127 T1548 oasc.CoreContainer.<init> New CoreContainer 520545337
[junit4:junit4] 2> 440128 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/'
[junit4:junit4] 2> 440128 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/'
[junit4:junit4] 2> 440170 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 440170 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 440170 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 440171 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 440171 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 440171 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 440172 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 440172 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 440172 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 440173 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 440176 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 440176 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 440176 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 440177 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 440179 T1588 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a43c347 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 440179 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 440181 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 440184 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 440185 T1590 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a8cb79c name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 440186 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 440189 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 441192 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59804_
[junit4:junit4] 2> 441194 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59804_
[junit4:junit4] 2> 441197 T1570 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 441197 T1590 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 441198 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 441197 T1577 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 441204 T1591 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 441204 T1591 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 441515 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 441516 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:58609__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58609_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58609"}
[junit4:junit4] 2> 441520 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59804_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59804"}
[junit4:junit4] 2> 441520 T1571 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 441521 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 441526 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 441526 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 441526 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 442206 T1591 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/collection1
[junit4:junit4] 2> 442206 T1591 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 442207 T1591 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 442207 T1591 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 442209 T1591 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/collection1/'
[junit4:junit4] 2> 442209 T1591 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/collection1/lib/README' to classloader
[junit4:junit4] 2> 442210 T1591 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 442234 T1591 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 442262 T1591 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 442263 T1591 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 442267 T1591 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 442579 T1591 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 442585 T1591 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 442587 T1591 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 442600 T1591 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 442603 T1591 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 442606 T1591 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 442607 T1591 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 442607 T1591 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 442608 T1591 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 442609 T1591 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 442609 T1591 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 442609 T1591 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 442610 T1591 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371582250453/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/
[junit4:junit4] 2> 442610 T1591 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 442610 T1591 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 442610 T1591 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1
[junit4:junit4] 2> 442611 T1591 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index/
[junit4:junit4] 2> 442611 T1591 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 442611 T1591 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index
[junit4:junit4] 2> 442614 T1591 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 442615 T1591 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 442616 T1591 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 442617 T1591 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 442617 T1591 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 442618 T1591 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 442618 T1591 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 442618 T1591 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 442619 T1591 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 442619 T1591 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 442619 T1591 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 442622 T1591 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 442624 T1591 oass.SolrIndexSearcher.<init> Opening Searcher@5b26e12e main
[junit4:junit4] 2> 442625 T1591 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 442625 T1591 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 442629 T1592 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5b26e12e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 442631 T1591 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 442631 T1591 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59804 collection:collection1 shard:shard1
[junit4:junit4] 2> 442632 T1591 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 442638 T1591 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 442640 T1591 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 442640 T1591 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 442640 T1591 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59804/collection1/
[junit4:junit4] 2> 442641 T1591 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 442641 T1591 oasc.SyncStrategy.syncToMe http://127.0.0.1:59804/collection1/ has no replicas
[junit4:junit4] 2> 442641 T1591 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59804/collection1/
[junit4:junit4] 2> 442641 T1591 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 443033 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 443042 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 443042 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 443042 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 443047 T1591 oasc.ZkController.register We are http://127.0.0.1:59804/collection1/ and leader is http://127.0.0.1:59804/collection1/
[junit4:junit4] 2> 443047 T1591 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59804
[junit4:junit4] 2> 443048 T1591 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 443048 T1591 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 443048 T1591 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 443049 T1591 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 443051 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 443051 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 443052 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 443126 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 443128 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60403
[junit4:junit4] 2> 443129 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 443129 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 443130 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457
[junit4:junit4] 2> 443130 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/solr.xml
[junit4:junit4] 2> 443130 T1548 oasc.CoreContainer.<init> New CoreContainer 531339176
[junit4:junit4] 2> 443131 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/'
[junit4:junit4] 2> 443131 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/'
[junit4:junit4] 2> 443174 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 443175 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 443175 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 443176 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 443176 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 443176 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 443177 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 443177 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 443177 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 443178 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 443181 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 443181 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 443181 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 443182 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443184 T1604 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36e9c7d8 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443185 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443186 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 443189 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443190 T1606 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@684d2953 name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443190 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443194 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 444197 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60403_
[junit4:junit4] 2> 444198 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60403_
[junit4:junit4] 2> 444201 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 444201 T1606 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 444202 T1570 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 444202 T1590 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 444202 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 444202 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 444203 T1577 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 444207 T1607 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 444207 T1607 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 444548 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 444549 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:59804__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59804_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59804"}
[junit4:junit4] 2> 444553 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60403_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60403"}
[junit4:junit4] 2> 444553 T1571 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 444553 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 444558 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 444558 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 444558 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 444559 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 445209 T1607 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/collection1
[junit4:junit4] 2> 445209 T1607 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 445210 T1607 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 445210 T1607 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 445211 T1607 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/collection1/'
[junit4:junit4] 2> 445211 T1607 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/collection1/lib/README' to classloader
[junit4:junit4] 2> 445212 T1607 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 445235 T1607 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 445263 T1607 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 445265 T1607 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 445268 T1607 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 445586 T1607 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 445592 T1607 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 445594 T1607 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 445605 T1607 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 445608 T1607 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 445611 T1607 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 445612 T1607 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 445612 T1607 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 445612 T1607 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 445613 T1607 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 445614 T1607 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 445614 T1607 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 445614 T1607 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371582253457/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/
[junit4:junit4] 2> 445614 T1607 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 445615 T1607 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 445615 T1607 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2
[junit4:junit4] 2> 445615 T1607 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index/
[junit4:junit4] 2> 445616 T1607 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 445616 T1607 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index
[junit4:junit4] 2> 445619 T1607 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 445619 T1607 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 445621 T1607 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 445621 T1607 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 445621 T1607 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 445622 T1607 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 445622 T1607 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 445622 T1607 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 445623 T1607 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 445623 T1607 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 445623 T1607 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 445626 T1607 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 445628 T1607 oass.SolrIndexSearcher.<init> Opening Searcher@6faf6609 main
[junit4:junit4] 2> 445629 T1607 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 445629 T1607 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 445633 T1608 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6faf6609 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 445634 T1607 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 445634 T1607 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60403 collection:collection1 shard:shard2
[junit4:junit4] 2> 445635 T1607 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 445640 T1607 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 445643 T1607 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 445643 T1607 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 445643 T1607 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60403/collection1/
[junit4:junit4] 2> 445643 T1607 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 445643 T1607 oasc.SyncStrategy.syncToMe http://127.0.0.1:60403/collection1/ has no replicas
[junit4:junit4] 2> 445643 T1607 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60403/collection1/
[junit4:junit4] 2> 445644 T1607 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 446065 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446071 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 446071 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 446071 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 446071 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 446099 T1607 oasc.ZkController.register We are http://127.0.0.1:60403/collection1/ and leader is http://127.0.0.1:60403/collection1/
[junit4:junit4] 2> 446099 T1607 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60403
[junit4:junit4] 2> 446099 T1607 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 446100 T1607 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 446100 T1607 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 446102 T1607 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446103 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 446103 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 446104 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 446167 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 446170 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49289
[junit4:junit4] 2> 446170 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 446170 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 446171 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508
[junit4:junit4] 2> 446171 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/solr.xml
[junit4:junit4] 2> 446171 T1548 oasc.CoreContainer.<init> New CoreContainer 311843898
[junit4:junit4] 2> 446172 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/'
[junit4:junit4] 2> 446172 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/'
[junit4:junit4] 2> 446212 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 446213 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 446213 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 446214 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 446214 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 446214 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 446215 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 446215 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 446215 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 446216 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 446218 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 446219 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 446219 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 446220 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446222 T1620 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dbea541 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446223 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446224 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 446227 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446228 T1622 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f30726f name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446229 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446232 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 447235 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49289_
[junit4:junit4] 2> 447237 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49289_
[junit4:junit4] 2> 447239 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447239 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447241 T1622 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 447241 T1590 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 447241 T1570 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 447241 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 447241 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 447243 T1577 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 447243 T1606 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 447246 T1623 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 447247 T1623 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 447576 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 447577 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:60403__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60403_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60403"}
[junit4:junit4] 2> 447580 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49289_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49289"}
[junit4:junit4] 2> 447580 T1571 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 447581 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 447584 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 447584 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 447584 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 447584 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 447584 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 448248 T1623 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/collection1
[junit4:junit4] 2> 448248 T1623 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 448249 T1623 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 448249 T1623 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 448250 T1623 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/collection1/'
[junit4:junit4] 2> 448251 T1623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/collection1/lib/README' to classloader
[junit4:junit4] 2> 448251 T1623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 448275 T1623 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 448303 T1623 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 448304 T1623 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 448307 T1623 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 448629 T1623 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 448634 T1623 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 448637 T1623 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 448648 T1623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 448651 T1623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 448653 T1623 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 448654 T1623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 448655 T1623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 448655 T1623 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 448656 T1623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 448656 T1623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 448656 T1623 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 448657 T1623 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371582256508/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty3/
[junit4:junit4] 2> 448657 T1623 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 448657 T1623 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 448658 T1623 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty3
[junit4:junit4] 2> 448658 T1623 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty3/index/
[junit4:junit4] 2> 448658 T1623 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 448658 T1623 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty3/index
[junit4:junit4] 2> 448661 T1623 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 448661 T1623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 448663 T1623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 448663 T1623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 448664 T1623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 448664 T1623 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 448665 T1623 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 448665 T1623 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 448665 T1623 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 448666 T1623 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 448666 T1623 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 448668 T1623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 448671 T1623 oass.SolrIndexSearcher.<init> Opening Searcher@2c41fe20 main
[junit4:junit4] 2> 448671 T1623 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 448672 T1623 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 448676 T1624 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c41fe20 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 448679 T1623 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 448679 T1623 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49289 collection:collection1 shard:shard3
[junit4:junit4] 2> 448680 T1623 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 448685 T1623 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 448687 T1623 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 448687 T1623 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 448687 T1623 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49289/collection1/
[junit4:junit4] 2> 448688 T1623 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 448688 T1623 oasc.SyncStrategy.syncToMe http://127.0.0.1:49289/collection1/ has no replicas
[junit4:junit4] 2> 448688 T1623 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49289/collection1/
[junit4:junit4] 2> 448689 T1623 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 449091 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449098 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 449098 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 449099 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 449098 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 449098 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 449144 T1623 oasc.ZkController.register We are http://127.0.0.1:49289/collection1/ and leader is http://127.0.0.1:49289/collection1/
[junit4:junit4] 2> 449145 T1623 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49289
[junit4:junit4] 2> 449145 T1623 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 449145 T1623 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 449145 T1623 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 449147 T1623 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449148 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 449148 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 449149 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 449218 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 449221 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40749
[junit4:junit4] 2> 449221 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 449222 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 449222 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553
[junit4:junit4] 2> 449222 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/solr.xml
[junit4:junit4] 2> 449223 T1548 oasc.CoreContainer.<init> New CoreContainer 242429267
[junit4:junit4] 2> 449223 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/'
[junit4:junit4] 2> 449223 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/'
[junit4:junit4] 2> 449267 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 449268 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 449268 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 449268 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 449269 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 449269 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 449269 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 449270 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 449270 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 449270 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 449273 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 449274 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 449274 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 449275 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449277 T1636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3415cd0c name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 449277 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 449279 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 449282 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449284 T1638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a7210fc name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 449284 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 449287 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 450290 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40749_
[junit4:junit4] 2> 450291 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40749_
[junit4:junit4] 2> 450294 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450294 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450295 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450296 T1570 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 450296 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450296 T1590 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 450296 T1638 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 450297 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450297 T1577 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 450297 T1606 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 450298 T1622 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 450301 T1639 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 450301 T1639 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 450604 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 450605 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49289__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49289_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49289"}
[junit4:junit4] 2> 450609 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40749_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40749"}
[junit4:junit4] 2> 450609 T1571 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 450609 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 450613 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450613 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450613 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450613 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450613 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 450613 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 451302 T1639 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/collection1
[junit4:junit4] 2> 451303 T1639 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 451303 T1639 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 451304 T1639 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 451305 T1639 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/collection1/'
[junit4:junit4] 2> 451305 T1639 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/collection1/lib/README' to classloader
[junit4:junit4] 2> 451306 T1639 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 451329 T1639 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 451357 T1639 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 451358 T1639 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 451362 T1639 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 451653 T1639 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 451658 T1639 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 451661 T1639 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 451672 T1639 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 451675 T1639 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 451699 T1639 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 451701 T1639 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 451701 T1639 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 451701 T1639 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 451702 T1639 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 451702 T1639 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 451703 T1639 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 451703 T1639 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371582259553/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/
[junit4:junit4] 2> 451703 T1639 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 451704 T1639 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 451704 T1639 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4
[junit4:junit4] 2> 451704 T1639 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index/
[junit4:junit4] 2> 451704 T1639 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 451705 T1639 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index
[junit4:junit4] 2> 451707 T1639 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 451708 T1639 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 451709 T1639 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 451710 T1639 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 451710 T1639 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 451711 T1639 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 451711 T1639 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 451711 T1639 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 451712 T1639 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 451712 T1639 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 451712 T1639 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 451715 T1639 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 451717 T1639 oass.SolrIndexSearcher.<init> Opening Searcher@690545ce main
[junit4:junit4] 2> 451718 T1639 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 451718 T1639 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 451722 T1640 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@690545ce main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 451723 T1639 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 451723 T1639 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40749 collection:collection1 shard:shard1
[junit4:junit4] 2> 451726 T1639 oasc.ZkController.register We are http://127.0.0.1:40749/collection1/ and leader is http://127.0.0.1:59804/collection1/
[junit4:junit4] 2> 451726 T1639 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40749
[junit4:junit4] 2> 451727 T1639 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 451727 T1639 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1373 name=collection1 org.apache.solr.core.SolrCore@33d47cfd url=http://127.0.0.1:40749/collection1 node=127.0.0.1:40749_ C1373_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:40749_, base_url=http://127.0.0.1:40749}
[junit4:junit4] 2> 451727 T1641 C1373 P40749 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 451728 T1639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 451728 T1641 C1373 P40749 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 451729 T1641 C1373 P40749 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 451729 T1641 C1373 P40749 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 451730 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 451730 T1641 C1373 P40749 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 451730 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 451731 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 451735 T1585 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:40749__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 451799 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 451802 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47760
[junit4:junit4] 2> 451802 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 451803 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 451803 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135
[junit4:junit4] 2> 451804 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/solr.xml
[junit4:junit4] 2> 451804 T1548 oasc.CoreContainer.<init> New CoreContainer 192544762
[junit4:junit4] 2> 451805 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/'
[junit4:junit4] 2> 451805 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/'
[junit4:junit4] 2> 451846 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 451847 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 451847 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 451848 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 451848 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 451848 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 451849 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 451849 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 451850 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 451850 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 451853 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 451853 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 451853 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 451854 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 451856 T1653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@377c5d33 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 451856 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 451858 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 451861 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 451863 T1655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33bedc11 name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 451863 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 451866 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 452120 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452120 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40749__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40749_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40749"}
[junit4:junit4] 2> 452131 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452131 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452131 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452131 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452131 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452131 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452131 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452736 T1585 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:40749__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 452737 T1585 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:40749__collection1&state=recovering&nodeName=127.0.0.1:40749_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 452869 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47760_
[junit4:junit4] 2> 452871 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47760_
[junit4:junit4] 2> 452873 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452873 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452873 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452873 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 452873 T1638 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452874 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 452873 T1590 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452874 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 452874 T1570 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452875 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 452875 T1606 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452876 T1577 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452876 T1655 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452876 T1622 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 452881 T1656 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 452881 T1656 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 453638 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 453639 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47760_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47760"}
[junit4:junit4] 2> 453639 T1571 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 453639 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 453644 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453644 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453644 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453644 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453644 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453644 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453644 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 453882 T1656 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/collection1
[junit4:junit4] 2> 453883 T1656 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 453883 T1656 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 453883 T1656 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 453885 T1656 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/collection1/'
[junit4:junit4] 2> 453885 T1656 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/collection1/lib/README' to classloader
[junit4:junit4] 2> 453886 T1656 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 453909 T1656 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 453937 T1656 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 453939 T1656 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 453942 T1656 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 454235 T1656 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 454240 T1656 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 454243 T1656 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 454255 T1656 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 454258 T1656 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 454260 T1656 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 454262 T1656 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 454262 T1656 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 454262 T1656 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 454263 T1656 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 454263 T1656 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 454263 T1656 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 454264 T1656 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371582262135/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/
[junit4:junit4] 2> 454264 T1656 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 454264 T1656 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 454265 T1656 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5
[junit4:junit4] 2> 454265 T1656 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index/
[junit4:junit4] 2> 454266 T1656 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 454266 T1656 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index
[junit4:junit4] 2> 454269 T1656 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 454269 T1656 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 454271 T1656 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 454271 T1656 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 454272 T1656 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 454272 T1656 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 454273 T1656 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 454273 T1656 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 454273 T1656 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 454274 T1656 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 454274 T1656 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 454276 T1656 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 454279 T1656 oass.SolrIndexSearcher.<init> Opening Searcher@22907ab9 main
[junit4:junit4] 2> 454280 T1656 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 454280 T1656 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 454284 T1657 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22907ab9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 454286 T1656 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 454286 T1656 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47760 collection:collection1 shard:shard2
[junit4:junit4] 2> 454289 T1656 oasc.ZkController.register We are http://127.0.0.1:47760/collection1/ and leader is http://127.0.0.1:60403/collection1/
[junit4:junit4] 2> 454289 T1656 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47760
[junit4:junit4] 2> 454289 T1656 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 454290 T1656 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1374 name=collection1 org.apache.solr.core.SolrCore@717165f7 url=http://127.0.0.1:47760/collection1 node=127.0.0.1:47760_ C1374_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47760_, base_url=http://127.0.0.1:47760}
[junit4:junit4] 2> 454290 T1658 C1374 P47760 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 454290 T1658 C1374 P47760 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 454290 T1656 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 454291 T1658 C1374 P47760 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 454291 T1658 C1374 P47760 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 454291 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 454292 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 454292 T1658 C1374 P47760 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 454292 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 454296 T1601 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:47760__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 454391 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 454393 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36199
[junit4:junit4] 2> 454394 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 454394 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 454395 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697
[junit4:junit4] 2> 454395 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/solr.xml
[junit4:junit4] 2> 454395 T1548 oasc.CoreContainer.<init> New CoreContainer 1874700893
[junit4:junit4] 2> 454396 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/'
[junit4:junit4] 2> 454396 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/'
[junit4:junit4] 2> 454466 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 454467 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 454467 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 454467 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 454468 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 454468 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 454468 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 454469 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 454469 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 454469 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 454472 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 454473 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 454473 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 454474 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 454476 T1670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@386d06b6 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 454476 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 454478 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 454481 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 454482 T1672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@324f7ef8 name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 454482 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 454486 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1375 name=collection1 org.apache.solr.core.SolrCore@33d47cfd url=http://127.0.0.1:40749/collection1 node=127.0.0.1:40749_ C1375_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:40749_, base_url=http://127.0.0.1:40749}
[junit4:junit4] 2> 454737 T1641 C1375 P40749 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59804/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 454738 T1641 C1375 P40749 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40749 START replicas=[http://127.0.0.1:59804/collection1/] nUpdates=100
[junit4:junit4] 2> 454738 T1641 C1375 P40749 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 454739 T1641 C1375 P40749 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 454739 T1641 C1375 P40749 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 454739 T1641 C1375 P40749 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 454739 T1641 C1375 P40749 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 454739 T1641 C1375 P40749 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59804/collection1/. core=collection1
[junit4:junit4] 2> 454739 T1641 C1375 P40749 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1376 name=collection1 org.apache.solr.core.SolrCore@37f11225 url=http://127.0.0.1:59804/collection1 node=127.0.0.1:59804_ C1376_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59804_, base_url=http://127.0.0.1:59804, leader=true}
[junit4:junit4] 2> 454744 T1586 C1376 P59804 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 454746 T1583 C1376 P59804 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 454749 T1583 C1376 P59804 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 454749 T1583 C1376 P59804 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 454755 T1583 C1376 P59804 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 454755 T1583 C1376 P59804 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 454756 T1583 C1376 P59804 oass.SolrIndexSearcher.<init> Opening Searcher@1997ebc8 realtime
[junit4:junit4] 2> 454756 T1583 C1376 P59804 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 454757 T1583 C1376 P59804 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 454757 T1641 C1375 P40749 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 454758 T1641 C1375 P40749 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 454759 T1585 C1376 P59804 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 454760 T1585 C1376 P59804 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 454760 T1641 C1375 P40749 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 454760 T1641 C1375 P40749 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 454760 T1641 C1375 P40749 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 454762 T1586 C1376 P59804 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 454763 T1641 C1375 P40749 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 454764 T1641 C1375 P40749 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index.20130618150425164
[junit4:junit4] 2> 454764 T1641 C1375 P40749 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index.20130618150425164 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7b336ba1 fullCopy=false
[junit4:junit4] 2> 454766 T1583 C1376 P59804 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 454767 T1641 C1375 P40749 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 454775 T1641 C1375 P40749 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 454776 T1641 C1375 P40749 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 454777 T1641 C1375 P40749 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 454778 T1641 C1375 P40749 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 454778 T1641 C1375 P40749 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 454778 T1641 C1375 P40749 oass.SolrIndexSearcher.<init> Opening Searcher@19718f83 main
[junit4:junit4] 2> 454779 T1640 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19718f83 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 454780 T1641 C1375 P40749 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index.20130618150425164 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index.20130618150425164;done=true>>]
[junit4:junit4] 2> 454780 T1641 C1375 P40749 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index.20130618150425164
[junit4:junit4] 2> 454780 T1641 C1375 P40749 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty4/index.20130618150425164
[junit4:junit4] 2> 454780 T1641 C1375 P40749 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 454781 T1641 C1375 P40749 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 454781 T1641 C1375 P40749 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 454781 T1641 C1375 P40749 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 454783 T1641 C1375 P40749 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 455149 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 455150 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:47760__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47760_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47760"}
[junit4:junit4] 2> 455154 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40749__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40749_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40749"}
[junit4:junit4] 2> 455158 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455158 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455297 T1601 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:47760__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 455297 T1601 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:47760__collection1&state=recovering&nodeName=127.0.0.1:47760_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 455489 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36199_
[junit4:junit4] 2> 455491 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36199_
[junit4:junit4] 2> 455492 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455492 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455492 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455492 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455493 T1590 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455494 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 455493 T1638 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455493 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 455494 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 455494 T1570 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455495 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 455495 T1577 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455496 T1622 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455496 T1606 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455497 T1655 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455497 T1672 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 455502 T1675 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 455503 T1675 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 456666 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 456667 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36199_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36199"}
[junit4:junit4] 2> 456668 T1571 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 456668 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 456673 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 456673 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> ASYNC NEW_CORE C1377 name=collection1 org.apache.solr.core.SolrCore@717165f7 url=http://127.0.0.1:47760/collection1 node=127.0.0.1:47760_ C1377_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:47760_, base_url=http://127.0.0.1:47760}
[junit4:junit4] 2> 457298 T1658 C1377 P47760 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:60403/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 457298 T1658 C1377 P47760 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47760 START replicas=[http://127.0.0.1:60403/collection1/] nUpdates=100
[junit4:junit4] 2> 457299 T1658 C1377 P47760 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 457300 T1658 C1377 P47760 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 457300 T1658 C1377 P47760 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 457300 T1658 C1377 P47760 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 457300 T1658 C1377 P47760 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 457300 T1658 C1377 P47760 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:60403/collection1/. core=collection1
[junit4:junit4] 2> 457301 T1658 C1377 P47760 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1378 name=collection1 org.apache.solr.core.SolrCore@8ea9781 url=http://127.0.0.1:60403/collection1 node=127.0.0.1:60403_ C1378_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:60403_, base_url=http://127.0.0.1:60403, leader=true}
[junit4:junit4] 2> 457304 T1601 C1378 P60403 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 457306 T1600 C1378 P60403 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 457308 T1600 C1378 P60403 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 457308 T1600 C1378 P60403 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 457315 T1600 C1378 P60403 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 457316 T1600 C1378 P60403 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 457316 T1600 C1378 P60403 oass.SolrIndexSearcher.<init> Opening Searcher@51b0257b realtime
[junit4:junit4] 2> 457316 T1600 C1378 P60403 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 457317 T1600 C1378 P60403 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 457318 T1658 C1377 P47760 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 457318 T1658 C1377 P47760 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 457319 T1601 C1378 P60403 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 457319 T1601 C1378 P60403 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 457320 T1658 C1377 P47760 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 457320 T1658 C1377 P47760 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 457320 T1658 C1377 P47760 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 457322 T1599 C1378 P60403 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 457323 T1658 C1377 P47760 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 457323 T1658 C1377 P47760 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index.20130618150427724
[junit4:junit4] 2> 457323 T1658 C1377 P47760 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index.20130618150427724 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5539eff0 fullCopy=false
[junit4:junit4] 2> 457326 T1600 C1378 P60403 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 457327 T1658 C1377 P47760 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 457335 T1658 C1377 P47760 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 457336 T1658 C1377 P47760 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 457338 T1658 C1377 P47760 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 457338 T1658 C1377 P47760 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 457338 T1658 C1377 P47760 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 457339 T1658 C1377 P47760 oass.SolrIndexSearcher.<init> Opening Searcher@1f95fd5a main
[junit4:junit4] 2> 457340 T1657 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f95fd5a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 457340 T1658 C1377 P47760 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index.20130618150427724 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index.20130618150427724;done=true>>]
[junit4:junit4] 2> 457340 T1658 C1377 P47760 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index.20130618150427724
[junit4:junit4] 2> 457340 T1658 C1377 P47760 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty5/index.20130618150427724
[junit4:junit4] 2> 457341 T1658 C1377 P47760 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 457341 T1658 C1377 P47760 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 457341 T1658 C1377 P47760 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 457341 T1658 C1377 P47760 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 457343 T1658 C1377 P47760 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 457504 T1675 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/collection1
[junit4:junit4] 2> 457504 T1675 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 457505 T1675 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 457505 T1675 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 457507 T1675 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/collection1/'
[junit4:junit4] 2> 457507 T1675 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/collection1/lib/README' to classloader
[junit4:junit4] 2> 457507 T1675 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 457534 T1675 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 457562 T1675 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 457563 T1675 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 457566 T1675 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 457858 T1675 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 457863 T1675 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 457866 T1675 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 457877 T1675 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 457881 T1675 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 457883 T1675 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 457884 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 457885 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 457885 T1675 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 457886 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 457886 T1675 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 457886 T1675 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 457886 T1675 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371582264697/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty6/
[junit4:junit4] 2> 457887 T1675 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 457887 T1675 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 457887 T1675 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty6
[junit4:junit4] 2> 457888 T1675 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty6/index/
[junit4:junit4] 2> 457888 T1675 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 457888 T1675 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty6/index
[junit4:junit4] 2> 457907 T1675 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 457907 T1675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 457909 T1675 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 457909 T1675 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 457910 T1675 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 457910 T1675 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 457911 T1675 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 457911 T1675 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 457912 T1675 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 457913 T1675 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 457913 T1675 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 457915 T1675 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 457918 T1675 oass.SolrIndexSearcher.<init> Opening Searcher@4428fdd9 main
[junit4:junit4] 2> 457918 T1675 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 457918 T1675 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 457922 T1678 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4428fdd9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 457923 T1675 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 457924 T1675 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36199 collection:collection1 shard:shard3
[junit4:junit4] 2> 457927 T1675 oasc.ZkController.register We are http://127.0.0.1:36199/collection1/ and leader is http://127.0.0.1:49289/collection1/
[junit4:junit4] 2> 457927 T1675 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36199
[junit4:junit4] 2> 457927 T1675 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 457927 T1675 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1379 name=collection1 org.apache.solr.core.SolrCore@6e3610bc url=http://127.0.0.1:36199/collection1 node=127.0.0.1:36199_ C1379_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:36199_, base_url=http://127.0.0.1:36199}
[junit4:junit4] 2> 457928 T1679 C1379 P36199 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 457929 T1679 C1379 P36199 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 457929 T1675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 457929 T1679 C1379 P36199 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 457929 T1679 C1379 P36199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 457930 T1548 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 457931 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 457930 T1679 C1379 P36199 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 457931 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 457936 T1616 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:36199__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 457996 T1548 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 457998 T1548 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57606
[junit4:junit4] 2> 457998 T1548 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 457999 T1548 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 457999 T1548 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336
[junit4:junit4] 2> 458000 T1548 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/solr.xml
[junit4:junit4] 2> 458000 T1548 oasc.CoreContainer.<init> New CoreContainer 494652939
[junit4:junit4] 2> 458000 T1548 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/'
[junit4:junit4] 2> 458001 T1548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/'
[junit4:junit4] 2> 458061 T1548 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 458062 T1548 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 458062 T1548 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 458063 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 458063 T1548 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 458063 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 458064 T1548 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 458064 T1548 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 458064 T1548 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 458065 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 458067 T1548 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 458068 T1548 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42837/solr
[junit4:junit4] 2> 458068 T1548 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 458069 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 458071 T1691 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@515c1f73 name:ZooKeeperConnection Watcher:127.0.0.1:42837 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 458071 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 458073 T1548 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 458075 T1548 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 458077 T1693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d63bad1 name:ZooKeeperConnection Watcher:127.0.0.1:42837/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 458077 T1548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 458080 T1548 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 458179 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 458180 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:47760__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47760_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47760"}
[junit4:junit4] 2> 458183 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:36199__collection1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36199_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36199"}
[junit4:junit4] 2> 458190 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458191 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458190 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458937 T1616 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:36199__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 458937 T1616 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:36199__collection1&state=recovering&nodeName=127.0.0.1:36199_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 459084 T1548 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57606_
[junit4:junit4] 2> 459085 T1548 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57606_
[junit4:junit4] 2> 459087 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459087 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459087 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459088 T1570 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459087 T1693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459087 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459089 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459088 T1638 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459089 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459088 T1590 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459088 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459090 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459090 T1577 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459091 T1672 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459092 T1622 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459092 T1606 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459092 T1655 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459093 T1693 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 459096 T1694 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 459097 T1694 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 459697 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 459699 T1571 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57606_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57606"}
[junit4:junit4] 2> 459699 T1571 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 459699 T1571 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 459703 T1693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1590 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 459703 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 460098 T1694 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/collection1
[junit4:junit4] 2> 460098 T1694 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 460099 T1694 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 460099 T1694 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 460101 T1694 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/collection1/'
[junit4:junit4] 2> 460101 T1694 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/collection1/lib/README' to classloader
[junit4:junit4] 2> 460101 T1694 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 460125 T1694 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 460153 T1694 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 460154 T1694 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 460158 T1694 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 460479 T1694 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 460485 T1694 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 460488 T1694 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 460499 T1694 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 460502 T1694 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 460505 T1694 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 460507 T1694 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 460507 T1694 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 460507 T1694 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 460508 T1694 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 460508 T1694 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 460509 T1694 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 460509 T1694 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371582268336/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty7/
[junit4:junit4] 2> 460509 T1694 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8f66d63
[junit4:junit4] 2> 460510 T1694 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 460510 T1694 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty7
[junit4:junit4] 2> 460510 T1694 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty7/index/
[junit4:junit4] 2> 460511 T1694 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 460511 T1694 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty7/index
[junit4:junit4] 2> 460514 T1694 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371582246870/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 460514 T1694 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 460516 T1694 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 460516 T1694 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 460517 T1694 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 460517 T1694 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 460518 T1694 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 460518 T1694 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 460518 T1694 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 460519 T1694 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 460519 T1694 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 460522 T1694 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 460525 T1694 oass.SolrIndexSearcher.<init> Opening Searcher@3295d4f3 main
[junit4:junit4] 2> 460526 T1694 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 460526 T1694 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 460530 T1695 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3295d4f3 main{S
[...truncated too long message...]
cc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 974728 T2744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 974728 T2744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 974773 T2661 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 42692
[junit4:junit4] 2> 974774 T2661 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=580030057
[junit4:junit4] 2> 974912 T2683 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 974913 T2683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 974913 T2683 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 975775 T2661 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 975776 T2661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 975777 T2661 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1abdced7
[junit4:junit4] 2> 975783 T2661 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=56,cumulative_deletesById=40,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 975784 T2661 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 975784 T2661 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 975785 T2661 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 975786 T2661 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 975786 T2661 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 975787 T2661 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371582645374/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371582645374/jetty2/index;done=false>>]
[junit4:junit4] 2> 975787 T2661 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371582645374/jetty2/index
[junit4:junit4] 2> 975788 T2661 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371582645374/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371582645374/jetty2;done=false>>]
[junit4:junit4] 2> 975788 T2661 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371582645374/jetty2
[junit4:junit4] 2> 975789 T2756 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888040247361546-127.0.0.1:42692_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 975811 T2661 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 975866 T2661 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 975868 T2661 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47689 47689
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=650D64FB840D96FC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_LU -Dtests.timezone=SystemV/EST5 -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 141s J1 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<642> but was:<34>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([650D64FB840D96FC:E4EBEAE3F352F6C0]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 975884 T2661 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 140920 T2660 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4] 2> 976000 T2703 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 976001 T2703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 976001 T2703 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 977090 T2744 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 977091 T2744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 977091 T2744 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Memory doPackFST= true), text=PostingsFormat(name=Lucene41WithOrds), _version_=PostingsFormat(name=Memory doPackFST= true), rnd_b=PostingsFormat(name=SimpleText), intDefault=PostingsFormat(name=Memory doPackFST= true), timestamp=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=SimpleText), a_t=PostingsFormat(name=Memory doPackFST= true), range_facet_sl=PostingsFormat(name=Lucene41WithOrds), range_facet_si=PostingsFormat(name=Asserting), other_tl1=PostingsFormat(name=Memory doPackFST= true), multiDefault=PostingsFormat(name=Asserting), a_si=PostingsFormat(name=Asserting)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=de_LU, timezone=SystemV/EST5
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=2,free=225335784,total=536870912
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestMultiCoreConfBootstrap, TestCodecSupport, TestLMJelinekMercerSimilarityFactory, SpellCheckCollatorTest, TestSchemaNameResource, TestAnalyzedSuggestions, DebugComponentTest, SimplePostToolTest, TestAtomicUpdateErrorCases, TestCSVResponseWriter, FileUtilsTest, TestAddFieldRealTimeGet, TestCollationField, TestBinaryResponseWriter, UniqFieldsUpdateProcessorFactoryTest, TestDynamicFieldResource, DefaultValueUpdateProcessorTest, DisMaxRequestHandlerTest, FieldAnalysisRequestHandlerTest, BadIndexSchemaTest, TestDefaultSearchFieldResource, TestOmitPositions, SuggesterWFSTTest, URLClassifyProcessorTest, OpenCloseCoreStressTest, TestPartialUpdateDeduplication, UpdateParamsTest, TestStressReorder, OpenExchangeRatesOrgProviderTest, TestQueryTypes, IndexReaderFactoryTest, DistanceFunctionTest, BasicZkTest, DirectSolrSpellCheckerTest, TestLuceneMatchVersion, CurrencyFieldOpenExchangeTest, FullSolrCloudDistribCmdsTest, DocumentAnalysisRequestHandlerTest, ClusterStateTest, HardAutoCommitTest, JSONWriterTest, TestDistributedGrouping, TestHashPartitioner, TestCopyFieldCollectionResource, ShardRoutingCustomTest, TestSolrXMLSerializer, LeaderElectionIntegrationTest, TestFoldingMultitermQuery, DocValuesMultiTest, ConvertedLegacyTest, TestCSVLoader, TestJmxIntegration, TestSolrQueryParserDefaultOperatorResource, TestRecovery, TestUtils, StandardRequestHandlerTest, TestFiltering, SolrCoreCheckLockOnStartupTest, TestFieldTypeResource, TestSolrXml, XsltUpdateRequestHandlerTest, XmlUpdateRequestHandlerTest, TermVectorComponentDistributedTest, TestRangeQuery, SignatureUpdateProcessorFactoryTest, DateMathParserTest, HighlighterConfigTest, SolrCmdDistributorTest, TestDistributedSearch, TestCoreDiscovery, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, RecoveryZkTest]
[junit4:junit4] Completed on J1 in 142.20s, 1 test, 1 failure <<< FAILURES!
[...truncated 449 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1250 tests, 1 suite-level error, 1 failure, 13 ignored (7 assumptions)
Total time: 41 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure