You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/22 16:40:25 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6256 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6256/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:39577/gi_/collection1lastClient and got 265 from http://127.0.0.1:35494/gi_/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:39577/gi_/collection1lastClient and got 265 from http://127.0.0.1:35494/gi_/collection1
at __randomizedtesting.SeedInfo.seed([9663CFCF20185844:178541D757473878]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9207 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
[junit4:junit4] 2> 835 T11 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /gi_/
[junit4:junit4] 2> 870 T11 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371910825341
[junit4:junit4] 2> 874 T11 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 1345 T12 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 1477 T11 oasc.ZkTestServer.run start zk server on port:44818
[junit4:junit4] 2> 1534 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1771 T18 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ded0b93 name:ZooKeeperConnection Watcher:127.0.0.1:44818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1773 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1775 T11 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 1808 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1811 T20 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65d42975 name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1812 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1819 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 1826 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 1831 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 1836 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 1850 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1875 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1888 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 1890 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 1899 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1900 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1907 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 1909 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 1916 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 1917 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 1924 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1926 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1933 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1934 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1941 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1943 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1949 T11 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1951 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 2310 T11 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2397 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45254
[junit4:junit4] 2> 2413 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2434 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2435 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428
[junit4:junit4] 2> 2444 T11 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/solr.xml
[junit4:junit4] 2> 2450 T11 oasc.CoreContainer.<init> New CoreContainer 2046984365
[junit4:junit4] 2> 2451 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/'
[junit4:junit4] 2> 2452 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/'
[junit4:junit4] 2> 2746 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2747 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2747 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2748 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2748 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2749 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2750 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2750 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2751 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2770 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3101 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 3105 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44818/solr
[junit4:junit4] 2> 3119 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3121 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3146 T33 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@120502b2 name:ZooKeeperConnection Watcher:127.0.0.1:44818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3147 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3152 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3162 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3164 T35 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10a13ab8 name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3165 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3172 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 3185 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 3207 T11 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 3222 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45254_gi_
[junit4:junit4] 2> 3226 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45254_gi_
[junit4:junit4] 2> 3241 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 3260 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 3270 T11 oasc.Overseer.start Overseer (id=89909547888934915-127.0.0.1:45254_gi_-n_0000000000) starting
[junit4:junit4] 2> 3286 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 3305 T37 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 3323 T11 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 3330 T11 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 3335 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3350 T36 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 3432 T38 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 3433 T38 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 4898 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4900 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45254_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45254/gi_"}
[junit4:junit4] 2> 4901 T36 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 4912 T36 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 4926 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 5437 T38 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/collection1
[junit4:junit4] 2> 5438 T38 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 5439 T38 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 5440 T38 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 5443 T38 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/collection1/'
[junit4:junit4] 2> 5445 T38 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/collection1/lib/README' to classloader
[junit4:junit4] 2> 5446 T38 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 5544 T38 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 5717 T38 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 5731 T38 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 5816 T38 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 6732 T38 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 6756 T38 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 6759 T38 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 6972 T38 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 6981 T38 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 6987 T38 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 6991 T38 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 6991 T38 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 6992 T38 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 6994 T38 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 6994 T38 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 7020 T38 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 7036 T38 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371910826428/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/
[junit4:junit4] 2> 7039 T38 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b0a66
[junit4:junit4] 2> 7081 T38 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 7106 T38 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data
[junit4:junit4] 2> 7107 T38 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index/
[junit4:junit4] 2> 7107 T38 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 7115 T38 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index
[junit4:junit4] 2> 7272 T38 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 7273 T38 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 7497 T38 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 7498 T38 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 7527 T38 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 7544 T38 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 7553 T38 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 7554 T38 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 7556 T38 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 7564 T38 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 7566 T38 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 7612 T38 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 7639 T38 oass.SolrIndexSearcher.<init> Opening Searcher@36205329 main
[junit4:junit4] 2> 7662 T38 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 7662 T38 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 7718 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36205329 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 7719 T38 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7720 T38 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45254/gi_ collection:control_collection shard:shard1
[junit4:junit4] 2> 7723 T38 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 7732 T38 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 7743 T38 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7746 T38 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7747 T38 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7748 T38 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45254/gi_/collection1/
[junit4:junit4] 2> 7749 T38 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7749 T38 oasc.SyncStrategy.syncToMe http://127.0.0.1:45254/gi_/collection1/ has no replicas
[junit4:junit4] 2> 7749 T38 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45254/gi_/collection1/
[junit4:junit4] 2> 7750 T38 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 7937 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7950 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 7960 T38 oasc.ZkController.register We are http://127.0.0.1:45254/gi_/collection1/ and leader is http://127.0.0.1:45254/gi_/collection1/
[junit4:junit4] 2> 7961 T38 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45254/gi_
[junit4:junit4] 2> 7962 T38 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7962 T38 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 7962 T38 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7965 T38 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7969 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 7969 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 7983 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 7992 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 7996 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 7998 T42 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@586efe67 name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 7998 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8001 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 8007 T11 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 8106 T11 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 8109 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39577
[junit4:junit4] 2> 8110 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 8111 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 8112 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477
[junit4:junit4] 2> 8113 T11 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/solr.xml
[junit4:junit4] 2> 8113 T11 oasc.CoreContainer.<init> New CoreContainer 1030380728
[junit4:junit4] 2> 8114 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/'
[junit4:junit4] 2> 8115 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/'
[junit4:junit4] 2> 8228 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8229 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8230 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8231 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8232 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8233 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8233 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8234 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8236 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8237 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8244 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 8245 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44818/solr
[junit4:junit4] 2> 8246 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8254 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8255 T53 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@146f47d8 name:ZooKeeperConnection Watcher:127.0.0.1:44818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8255 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8259 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8266 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8270 T55 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e17578f name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8271 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8278 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9285 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39577_gi_
[junit4:junit4] 2> 9287 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39577_gi_
[junit4:junit4] 2> 9292 T35 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 9292 T55 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 9292 T42 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 9293 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9303 T56 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9304 T56 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 9457 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9459 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45254_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45254_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45254/gi_"}
[junit4:junit4] 2> 9466 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39577_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39577/gi_"}
[junit4:junit4] 2> 9466 T36 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 9467 T36 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 9473 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9473 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9473 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10306 T56 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/collection1
[junit4:junit4] 2> 10307 T56 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 10308 T56 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 10308 T56 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 10310 T56 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/collection1/'
[junit4:junit4] 2> 10311 T56 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/collection1/lib/README' to classloader
[junit4:junit4] 2> 10312 T56 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 10355 T56 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 10414 T56 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 10417 T56 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 10469 T56 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 10929 T56 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 10943 T56 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 10945 T56 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 10984 T56 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 10992 T56 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 10999 T56 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 11002 T56 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 11002 T56 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 11003 T56 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 11005 T56 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 11005 T56 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 11006 T56 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 11006 T56 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371910832477/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/
[junit4:junit4] 2> 11007 T56 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b0a66
[junit4:junit4] 2> 11009 T56 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 11010 T56 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1
[junit4:junit4] 2> 11011 T56 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index/
[junit4:junit4] 2> 11012 T56 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 11013 T56 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index
[junit4:junit4] 2> 11037 T56 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 11037 T56 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 11041 T56 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 11042 T56 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 11043 T56 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 11045 T56 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 11047 T56 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 11047 T56 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 11048 T56 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 11050 T56 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 11051 T56 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 11055 T56 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 11059 T56 oass.SolrIndexSearcher.<init> Opening Searcher@470d9d75 main
[junit4:junit4] 2> 11060 T56 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 11060 T56 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 11079 T57 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@470d9d75 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 11085 T56 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11086 T56 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39577/gi_ collection:collection1 shard:shard1
[junit4:junit4] 2> 11087 T56 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 11110 T56 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 11116 T56 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 11117 T56 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 11117 T56 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 11118 T56 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 11119 T56 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/ has no replicas
[junit4:junit4] 2> 11119 T56 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 11120 T56 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 12486 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12499 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 12500 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 12499 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 12535 T56 oasc.ZkController.register We are http://127.0.0.1:39577/gi_/collection1/ and leader is http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 12536 T56 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39577/gi_
[junit4:junit4] 2> 12536 T56 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 12536 T56 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 12537 T56 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12697 T56 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12699 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 12700 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 12701 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 12807 T11 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 12810 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41267
[junit4:junit4] 2> 12811 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 12812 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 12812 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176
[junit4:junit4] 2> 12813 T11 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/solr.xml
[junit4:junit4] 2> 12813 T11 oasc.CoreContainer.<init> New CoreContainer 665037498
[junit4:junit4] 2> 12814 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/'
[junit4:junit4] 2> 12815 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/'
[junit4:junit4] 2> 12879 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 12879 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 12880 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 12880 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 12881 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 12881 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 12882 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 12883 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 12883 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 12884 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 12891 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 12892 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44818/solr
[junit4:junit4] 2> 12893 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 12895 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12898 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a7e89f2 name:ZooKeeperConnection Watcher:127.0.0.1:44818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12899 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12902 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 12909 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 12912 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6af06702 name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 12912 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 12934 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 13941 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41267_gi_
[junit4:junit4] 2> 13942 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41267_gi_
[junit4:junit4] 2> 13947 T35 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 13947 T71 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 13947 T42 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 13947 T55 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 13948 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13948 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13947 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13955 T72 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 13956 T72 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 14009 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14012 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:39577_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39577_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39577/gi_"}
[junit4:junit4] 2> 14018 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41267_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41267/gi_"}
[junit4:junit4] 2> 14018 T36 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 14019 T36 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 14025 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14025 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14025 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14025 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14958 T72 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/collection1
[junit4:junit4] 2> 14959 T72 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 14960 T72 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 14960 T72 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 14963 T72 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/collection1/'
[junit4:junit4] 2> 14964 T72 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/collection1/lib/README' to classloader
[junit4:junit4] 2> 14965 T72 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 15010 T72 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 15052 T72 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 15054 T72 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 15101 T72 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 15531 T72 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 15544 T72 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 15546 T72 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 15581 T72 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 15587 T72 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 15592 T72 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 15594 T72 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 15594 T72 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 15595 T72 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 15597 T72 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 15597 T72 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 15598 T72 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 15598 T72 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371910837176/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/
[junit4:junit4] 2> 15598 T72 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b0a66
[junit4:junit4] 2> 15599 T72 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 15600 T72 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2
[junit4:junit4] 2> 15600 T72 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index/
[junit4:junit4] 2> 15601 T72 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 15601 T72 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index
[junit4:junit4] 2> 15605 T72 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 15605 T72 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 15609 T72 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 15609 T72 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 15610 T72 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 15612 T72 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 15613 T72 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 15614 T72 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 15614 T72 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 15616 T72 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 15617 T72 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 15619 T72 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 15623 T72 oass.SolrIndexSearcher.<init> Opening Searcher@49b0cd6e main
[junit4:junit4] 2> 15623 T72 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 15624 T72 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 15639 T73 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49b0cd6e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 15645 T72 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 15646 T72 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41267/gi_ collection:collection1 shard:shard1
[junit4:junit4] 2> 15651 T72 oasc.ZkController.register We are http://127.0.0.1:41267/gi_/collection1/ and leader is http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 15651 T72 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41267/gi_
[junit4:junit4] 2> 15651 T72 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 15652 T72 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 15655 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15657 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> ASYNC NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@15cb79b7 url=http://127.0.0.1:41267/gi_/collection1 node=127.0.0.1:41267_gi_ C0_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41267_gi_, base_url=http://127.0.0.1:41267/gi_}
[junit4:junit4] 2> 15657 T74 C0 P41267 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 15658 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 15659 T74 C0 P41267 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 15659 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 15660 T74 C0 P41267 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 15661 T74 C0 P41267 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15663 T74 C0 P41267 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 15728 T11 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 15731 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35494
[junit4:junit4] 2> 15732 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 15733 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 15733 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133
[junit4:junit4] 2> 15733 T11 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/solr.xml
[junit4:junit4] 2> 15734 T11 oasc.CoreContainer.<init> New CoreContainer 1033073165
[junit4:junit4] 2> 15734 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/'
[junit4:junit4] 2> 15735 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/'
[junit4:junit4] 2> 15790 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 15791 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 15792 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 15792 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 15793 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 15793 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 15793 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 15794 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 15794 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 15795 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 15799 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 15800 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44818/solr
[junit4:junit4] 2> 15800 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 15801 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 15804 T86 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8198275 name:ZooKeeperConnection Watcher:127.0.0.1:44818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 15805 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 15807 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 15811 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 15814 T88 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b3fb1ab name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 15814 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 15822 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 15916 T48 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:41267_gi__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 16836 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35494_gi_
[junit4:junit4] 2> 16838 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35494_gi_
[junit4:junit4] 2> 16842 T35 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 16843 T88 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 16843 T71 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 16842 T42 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 16842 T55 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 16845 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16845 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16843 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16846 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 16852 T89 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 16853 T89 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 17040 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17041 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:41267_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41267_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41267/gi_"}
[junit4:junit4] 2> 17048 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35494_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35494/gi_"}
[junit4:junit4] 2> 17049 T36 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 17050 T36 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 17056 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17057 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17056 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17056 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17057 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 17855 T89 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/collection1
[junit4:junit4] 2> 17855 T89 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 17857 T89 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 17857 T89 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 17859 T89 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/collection1/'
[junit4:junit4] 2> 17860 T89 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/collection1/lib/README' to classloader
[junit4:junit4] 2> 17860 T89 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 17902 T89 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 17917 T48 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:41267_gi__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 17919 T48 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:41267_gi__collection1&state=recovering&nodeName=127.0.0.1:41267_gi_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2006
[junit4:junit4] 2> 17953 T89 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 17955 T89 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 17998 T89 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 18471 T89 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 18482 T89 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 18485 T89 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 18516 T89 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 18520 T89 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 18525 T89 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 18527 T89 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 18527 T89 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 18528 T89 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 18529 T89 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 18529 T89 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 18530 T89 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 18530 T89 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371910840133/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/
[junit4:junit4] 2> 18530 T89 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b0a66
[junit4:junit4] 2> 18531 T89 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 18532 T89 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3
[junit4:junit4] 2> 18532 T89 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index/
[junit4:junit4] 2> 18533 T89 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 18533 T89 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index
[junit4:junit4] 2> 18537 T89 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 18537 T89 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 18541 T89 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 18541 T89 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 18543 T89 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 18544 T89 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 18545 T89 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 18545 T89 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 18546 T89 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 18547 T89 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 18548 T89 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 18550 T89 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 18554 T89 oass.SolrIndexSearcher.<init> Opening Searcher@6778821e main
[junit4:junit4] 2> 18555 T89 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 18555 T89 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 18570 T90 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6778821e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 18577 T89 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 18577 T89 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35494/gi_ collection:collection1 shard:shard1
[junit4:junit4] 2> 18582 T89 oasc.ZkController.register We are http://127.0.0.1:35494/gi_/collection1/ and leader is http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 18582 T89 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35494/gi_
[junit4:junit4] 2> 18583 T89 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 18583 T89 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@61fa0030 url=http://127.0.0.1:35494/gi_/collection1 node=127.0.0.1:35494_gi_ C1_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:35494_gi_, base_url=http://127.0.0.1:35494/gi_}
[junit4:junit4] 2> 18584 T91 C1 P35494 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 18585 T91 C1 P35494 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 18585 T89 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18585 T91 C1 P35494 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 18586 T91 C1 P35494 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 18587 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 18588 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 18588 T91 C1 P35494 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 18588 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 18598 T51 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:35494_gi__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 18664 T11 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 18667 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45480
[junit4:junit4] 2> 18667 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 18668 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 18668 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063
[junit4:junit4] 2> 18669 T11 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/solr.xml
[junit4:junit4] 2> 18669 T11 oasc.CoreContainer.<init> New CoreContainer 401064227
[junit4:junit4] 2> 18670 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/'
[junit4:junit4] 2> 18670 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/'
[junit4:junit4] 2> 18724 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 18724 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 18725 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 18725 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 18726 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 18726 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 18726 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 18727 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 18727 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 18728 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 18732 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 18733 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44818/solr
[junit4:junit4] 2> 18733 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 18734 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 18736 T103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19a04ff3 name:ZooKeeperConnection Watcher:127.0.0.1:44818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 18737 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 18740 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 18744 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 18746 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38cd610f name:ZooKeeperConnection Watcher:127.0.0.1:44818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 18746 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 18752 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 19756 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45480_gi_
[junit4:junit4] 2> 19758 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45480_gi_
[junit4:junit4] 2> 19762 T88 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 19762 T35 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 19763 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 19762 T42 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 19762 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 19764 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 19763 T105 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 19763 T55 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 19763 T71 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 19765 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 19765 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 19774 T106 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 19774 T106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> ASYNC NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@15cb79b7 url=http://127.0.0.1:41267/gi_/collection1 node=127.0.0.1:41267_gi_ C2_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41267_gi_, base_url=http://127.0.0.1:41267/gi_}
[junit4:junit4] 2> 19975 T74 C2 P41267 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39577/gi_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 19978 T74 C2 P41267 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 19983 T74 C2 P41267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41267/gi_ START replicas=[http://127.0.0.1:39577/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 19986 T74 C2 P41267 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 19986 T74 C2 P41267 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 19986 T74 C2 P41267 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 19987 T74 C2 P41267 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 19987 T74 C2 P41267 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 19987 T74 C2 P41267 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39577/gi_/collection1/. core=collection1
[junit4:junit4] 2> 19988 T74 C2 P41267 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@611efcc2 url=http://127.0.0.1:39577/gi_/collection1 node=127.0.0.1:39577_gi_ C3_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:39577_gi_, base_url=http://127.0.0.1:39577/gi_, leader=true}
[junit4:junit4] 2> 20003 T49 C3 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4
[junit4:junit4] 2> 20022 T48 C3 P39577 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 20026 T48 C3 P39577 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 20027 T48 C3 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 20035 T48 C3 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 20036 T48 C3 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 20037 T48 C3 P39577 oass.SolrIndexSearcher.<init> Opening Searcher@225fd7c0 realtime
[junit4:junit4] 2> 20038 T48 C3 P39577 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 20038 T48 C3 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 36
[junit4:junit4] 2> 20054 T74 C2 P41267 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 20054 T74 C2 P41267 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 20060 T74 C2 P41267 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 20069 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 20070 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35494_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35494_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35494/gi_"}
[junit4:junit4] 2> 20075 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45480_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45480/gi_"}
[junit4:junit4] 2> 20076 T36 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 20076 T36 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 20079 T49 C3 P39577 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 20080 T49 C3 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=18
[junit4:junit4] 2> 20082 T74 C2 P41267 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 20082 T74 C2 P41267 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 20082 T74 C2 P41267 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 20086 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20086 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20086 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20086 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20086 T48 C3 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 20086 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20086 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 20090 T74 C2 P41267 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 20092 T74 C2 P41267 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index.20130622092044560
[junit4:junit4] 2> 20092 T74 C2 P41267 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index.20130622092044560 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60ce2f98; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 20099 T50 C3 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
[junit4:junit4] 2> 20104 T74 C2 P41267 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 20109 T74 C2 P41267 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 20110 T74 C2 P41267 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 20114 T74 C2 P41267 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 20115 T74 C2 P41267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 20115 T74 C2 P41267 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 20116 T74 C2 P41267 oass.SolrIndexSearcher.<init> Opening Searcher@2778a6a main
[junit4:junit4] 2> 20121 T73 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2778a6a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 20123 T74 C2 P41267 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index.20130622092044560 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index.20130622092044560;done=true>>]
[junit4:junit4] 2> 20123 T74 C2 P41267 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index.20130622092044560
[junit4:junit4] 2> ASYNC NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@15cb79b7 url=http://127.0.0.1:41267/gi_/collection1 node=127.0.0.1:41267_gi_ C4_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41267_gi_, base_url=http://127.0.0.1:41267/gi_}
[junit4:junit4] 2> 20128 T74 C4 P41267 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index.20130622092044560
[junit4:junit4] 2> 20131 T74 C4 P41267 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 20132 T74 C4 P41267 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 20132 T74 C4 P41267 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 20132 T74 C4 P41267 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 20134 T74 C4 P41267 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 20599 T51 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:35494_gi__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 20600 T51 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:35494_gi__collection1&state=recovering&nodeName=127.0.0.1:35494_gi_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4] 2> 20776 T106 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/collection1
[junit4:junit4] 2> 20776 T106 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 20778 T106 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 20778 T106 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 20780 T106 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/collection1/'
[junit4:junit4] 2> 20781 T106 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/collection1/lib/README' to classloader
[junit4:junit4] 2> 20782 T106 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 20821 T106 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 20867 T106 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 20869 T106 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 20901 T106 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 21299 T106 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 21310 T106 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 21313 T106 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 21343 T106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 21348 T106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 21364 T106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 21367 T106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 21367 T106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 21368 T106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 21369 T106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 21369 T106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 21370 T106 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 21370 T106 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371910843063/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/
[junit4:junit4] 2> 21370 T106 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c9b0a66
[junit4:junit4] 2> 21371 T106 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 21372 T106 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4
[junit4:junit4] 2> 21372 T106 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index/
[junit4:junit4] 2> 21373 T106 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 21373 T106 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index
[junit4:junit4] 2> 21388 T106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 21388 T106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 21392 T106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 21392 T106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 21393 T106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 21395 T106 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 21397 T106 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 21397 T106 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 21399 T106 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 21401 T106 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 21401 T106 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 21404 T106 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 21407 T106 oass.SolrIndexSearcher.<init> Opening Searcher@56a13ea6 main
[junit4:junit4] 2> 21408 T106 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 21409 T106 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 21420 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56a13ea6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 21424 T106 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 21425 T106 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45480/gi_ collection:collection1 shard:shard1
[junit4:junit4] 2> 21496 T106 oasc.ZkController.register We are http://127.0.0.1:45480/gi_/collection1/ and leader is http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 21496 T106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45480/gi_
[junit4:junit4] 2> 21497 T106 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 21497 T106 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@c1e18f2 url=http://127.0.0.1:45480/gi_/collection1 node=127.0.0.1:45480_gi_ C5_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:45480_gi_, base_url=http://127.0.0.1:45480/gi_}
[junit4:junit4] 2> 21498 T110 C5 P45480 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 21498 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 21499 T106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21499 T110 C5 P45480 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 21500 T110 C5 P45480 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 21501 T11 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 21501 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 21502 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 21512 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21512 T110 C5 P45480 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 21514 T11 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 21515 T11 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 21516 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21518 T49 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:45480_gi__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 21601 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21602 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:41267_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41267_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41267/gi_"}
[junit4:junit4] 2> 21607 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45480_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45480_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45480/gi_"}
[junit4:junit4] 2> 21613 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21614 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21614 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21613 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21613 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 21613 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 22518 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 22519 T49 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:45480_gi__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 22520 T49 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:45480_gi__collection1&state=recovering&nodeName=127.0.0.1:45480_gi_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> ASYNC NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@61fa0030 url=http://127.0.0.1:35494/gi_/collection1 node=127.0.0.1:35494_gi_ C6_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35494_gi_, base_url=http://127.0.0.1:35494/gi_}
[junit4:junit4] 2> 22601 T91 C6 P35494 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39577/gi_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 22602 T91 C6 P35494 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35494/gi_ START replicas=[http://127.0.0.1:39577/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 22603 T91 C6 P35494 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 22603 T91 C6 P35494 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 22604 T91 C6 P35494 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 22604 T91 C6 P35494 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 22604 T91 C6 P35494 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 22605 T91 C6 P35494 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39577/gi_/collection1/. core=collection1
[junit4:junit4] 2> 22606 T91 C6 P35494 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@611efcc2 url=http://127.0.0.1:39577/gi_/collection1 node=127.0.0.1:39577_gi_ C7_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:39577_gi_, base_url=http://127.0.0.1:39577/gi_, leader=true}
[junit4:junit4] 2> 22606 T51 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 22614 T48 C7 P39577 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 22626 T48 C7 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 22626 T48 C7 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 22627 T48 C7 P39577 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 22628 T48 C7 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 22629 T91 C6 P35494 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 22630 T91 C6 P35494 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 22634 T50 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 22635 T91 C6 P35494 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 22635 T91 C6 P35494 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 22636 T91 C6 P35494 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 22639 T49 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 22641 T91 C6 P35494 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 22642 T91 C6 P35494 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index.20130622092047110
[junit4:junit4] 2> 22642 T91 C6 P35494 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index.20130622092047110 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a305422; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 22647 T48 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=1
[junit4:junit4] 2> 22649 T91 C6 P35494 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 22651 T91 C6 P35494 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 22652 T91 C6 P35494 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 22655 T91 C6 P35494 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 22656 T91 C6 P35494 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 22656 T91 C6 P35494 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 22657 T91 C6 P35494 oass.SolrIndexSearcher.<init> Opening Searcher@5a7a87d1 main
[junit4:junit4] 2> 22660 T90 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a7a87d1 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 22661 T91 C6 P35494 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index.20130622092047110 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index.20130622092047110;done=true>>]
[junit4:junit4] 2> 22662 T91 C6 P35494 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index.20130622092047110
[junit4:junit4] 2> 22662 T91 C6 P35494 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index.20130622092047110
[junit4:junit4] 2> 22663 T91 C6 P35494 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 22663 T91 C6 P35494 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 22664 T91 C6 P35494 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 22664 T91 C6 P35494 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 22667 T91 C6 P35494 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 23125 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 23127 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35494_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35494_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35494/gi_"}
[junit4:junit4] 2> 23132 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23133 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23133 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23132 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23132 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23132 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23522 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C5_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:45480_gi_, base_url=http://127.0.0.1:45480/gi_}
[junit4:junit4] 2> 24521 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39577/gi_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 24522 T110 C5 P45480 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45480/gi_ START replicas=[http://127.0.0.1:39577/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 24522 T110 C5 P45480 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 24523 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 24523 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 24524 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 24524 T110 C5 P45480 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 24524 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24525 T110 C5 P45480 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39577/gi_/collection1/. core=collection1
[junit4:junit4] 2> 24526 T110 C5 P45480 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 24526 T51 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 24535 T49 C7 P39577 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 24546 T49 C7 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 24546 T49 C7 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 24547 T49 C7 P39577 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 24548 T49 C7 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 24549 T110 C5 P45480 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 24550 T110 C5 P45480 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 24553 T50 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 24555 T110 C5 P45480 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 24555 T110 C5 P45480 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 24555 T110 C5 P45480 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 24559 T49 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 24560 T110 C5 P45480 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 24561 T110 C5 P45480 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index.20130622092049030
[junit4:junit4] 2> 24562 T110 C5 P45480 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index.20130622092049030 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d0620b3; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 24565 T48 C7 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 24567 T110 C5 P45480 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 24570 T110 C5 P45480 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 24570 T110 C5 P45480 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 24575 T110 C5 P45480 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 24575 T110 C5 P45480 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 24576 T110 C5 P45480 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 24577 T110 C5 P45480 oass.SolrIndexSearcher.<init> Opening Searcher@666391ed main
[junit4:junit4] 2> 24579 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@666391ed main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 24581 T110 C5 P45480 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index.20130622092049030 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index.20130622092049030;done=true>>]
[junit4:junit4] 2> 24581 T110 C5 P45480 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index.20130622092049030
[junit4:junit4] 2> 24582 T110 C5 P45480 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index.20130622092049030
[junit4:junit4] 2> 24582 T110 C5 P45480 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 24583 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 24583 T110 C5 P45480 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 24584 T110 C5 P45480 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 24606 T110 C5 P45480 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 24641 T36 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24643 T36 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:45480_gi__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45480_gi_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45480/gi_"}
[junit4:junit4] 2> 24650 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24651 T35 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24650 T42 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24650 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24650 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24650 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25528 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25530 T11 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@4ca96e78 url=http://127.0.0.1:45254/gi_/collection1 node=127.0.0.1:45254_gi_ C8_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:45254_gi_, base_url=http://127.0.0.1:45254/gi_, leader=true}
[junit4:junit4] 2> 25536 T31 C8 P45254 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 25539 T31 C8 P45254 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 25540 T31 C8 P45254 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 25545 T31 C8 P45254 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 25546 T31 C8 P45254 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 25547 T31 C8 P45254 oass.SolrIndexSearcher.<init> Opening Searcher@3abfa5d0 main
[junit4:junit4] 2> 25547 T31 C8 P45254 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 25550 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3abfa5d0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 25551 T31 C8 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4] 2> C5_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45480_gi_, base_url=http://127.0.0.1:45480/gi_}
[junit4:junit4] 2> 25565 T98 C5 P45480 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39577/gi_/collection1/, StdNode: http://127.0.0.1:41267/gi_/collection1/, StdNode: http://127.0.0.1:35494/gi_/collection1/, StdNode: http://127.0.0.1:45480/gi_/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@611efcc2 url=http://127.0.0.1:39577/gi_/collection1 node=127.0.0.1:39577_gi_ C9_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:39577_gi_, base_url=http://127.0.0.1:39577/gi_, leader=true}
[junit4:junit4] 2> 25577 T51 C9 P39577 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@c1e18f2 url=http://127.0.0.1:45480/gi_/collection1 node=127.0.0.1:45480_gi_ C10_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45480_gi_, base_url=http://127.0.0.1:45480/gi_}
[junit4:junit4] 2> 25577 T99 C10 P45480 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@61fa0030 url=http://127.0.0.1:35494/gi_/collection1 node=127.0.0.1:35494_gi_ C11_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35494_gi_, base_url=http://127.0.0.1:35494/gi_}
[junit4:junit4] 2> 25582 T81 C11 P35494 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@15cb79b7 url=http://127.0.0.1:41267/gi_/collection1 node=127.0.0.1:41267_gi_ C12_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41267_gi_, base_url=http://127.0.0.1:41267/gi_}
[junit4:junit4] 2> 25584 T64 C12 P41267 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 25596 T51 C9 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 25596 T51 C9 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 25597 T51 C9 P39577 oass.SolrIndexSearcher.<init> Opening Searcher@3e1f3387 main
[junit4:junit4] 2> 25598 T51 C9 P39577 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 25599 T57 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3e1f3387 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 25600 T51 C9 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4] 2> 25626 T81 C11 P35494 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 25626 T64 C12 P41267 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 25626 T99 C10 P45480 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 25627 T64 C12 P41267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 25626 T81 C11 P35494 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 25628 T99 C10 P45480 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 25629 T81 C11 P35494 oass.SolrIndexSearcher.<init> Opening Searcher@3a300972 main
[junit4:junit4] 2> 25629 T64 C12 P41267 oass.SolrIndexSearcher.<init> Opening Searcher@29bd3793 main
[junit4:junit4] 2> 25630 T81 C11 P35494 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 25630 T99 C10 P45480 oass.SolrIndexSearcher.<init> Opening Searcher@689cf670 main
[junit4:junit4] 2> 25633 T73 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 25632 T90 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a300972 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 25631 T64 C12 P41267 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 25636 T73 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceAlreadyExistsException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 25637 T73 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29bd3793 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 25637 T64 C12 P41267 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 54
[junit4:junit4] 2> 25636 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@689cf670 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 25634 T99 C10 P45480 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 25636 T81 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 54
[junit4:junit4] 2> 25640 T99 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 63
[junit4:junit4] 2> 25641 T98 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 76
[junit4:junit4] 2> 25643 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25703 T49 C9 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=54
[junit4:junit4] 2> 25708 T66 C12 P41267 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 25711 T82 C11 P35494 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 25714 T101 C10 P45480 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> ASYNC NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@4ca96e78 url=http://127.0.0.1:45254/gi_/collection1 node=127.0.0.1:45254_gi_ C13_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:45254_gi_, base_url=http://127.0.0.1:45254/gi_, leader=true}
[junit4:junit4] 2> 27831 T28 C13 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438552793840353280)} 0 96
[junit4:junit4] 2> 27843 T81 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={update.distrib=FROMLEADER&_version_=-1438552793868664832&update.from=http://127.0.0.1:39577/gi_/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438552793868664832)} 0 1
[junit4:junit4] 2> 27844 T64 C12 P41267 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={update.distrib=FROMLEADER&_version_=-1438552793868664832&update.from=http://127.0.0.1:39577/gi_/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438552793868664832)} 0 2
[junit4:junit4] 2> 27845 T101 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={update.distrib=FROMLEADER&_version_=-1438552793868664832&update.from=http://127.0.0.1:39577/gi_/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438552793868664832)} 0 1
[junit4:junit4] 2> 27846 T50 C9 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438552793868664832)} 0 9
[junit4:junit4] 2> 27847 T66 C12 P41267 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 13
[junit4:junit4] 2> 27905 T31 C13 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {add=[0 (1438552793888587776)]} 0 53
[junit4:junit4] 2> 27927 T83 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438552793947308032)]} 0 6
[junit4:junit4] 2> 27930 T65 C12 P41267 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438552793947308032)]} 0 10
[junit4:junit4] 2> 27931 T100 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438552793947308032)]} 0 5
[junit4:junit4] 2> 27932 T48 C9 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:35494/gi_/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438552793947308032)]} 0 19
[junit4:junit4] 2> 27933 T84 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {add=[0]} 0 25
[junit4:junit4] 2> 27941 T30 C13 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {add=[1 (1438552793972473856)]} 0 4
[junit4:junit4] 2> 27962 T82 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438552793985056768)]} 0 6
[junit4:junit4] 2> 27962 T67 C12 P41267 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438552793985056768)]} 0 5
[junit4:junit4] 2> 27964 T99 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438552793985056768)]} 0 7
[junit4:junit4] 2> 27965 T51 C9 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:35494/gi_/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438552793985056768)]} 0 16
[junit4:junit4] 2> 27966 T81 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {add=[1]} 0 21
[junit4:junit4] 2> 27972 T29 C13 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {add=[2 (1438552794007076864)]} 0 3
[junit4:junit4] 2> 27981 T50 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:41267/gi_/collection1/]
[junit4:junit4] 2> 27982 T50 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:41267/gi_/collection1/ against:[http://127.0.0.1:41267/gi_/collection1/] result:true
[junit4:junit4] 2> 27982 T50 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35494/gi_/collection1/ against:[http://127.0.0.1:41267/gi_/collection1/] result:false
[junit4:junit4] 2> 27983 T50 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45480/gi_/collection1/ against:[http://127.0.0.1:41267/gi_/collection1/] result:false
[junit4:junit4] 2> 27995 T84 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[2 (1438552794020708352)]} 0 5
[junit4:junit4] 2> 27995 T98 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[2 (1438552794020708352)]} 0 5
[junit4:junit4] 2> 27997 T50 C9 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:35494/gi_/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[2 (1438552794020708352)]} 0 17
[junit4:junit4] 2> 27998 T83 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[2]} 0 22
[junit4:junit4] 2> 28006 T31 C13 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={wt=javabin&version=2} {add=[3 (1438552794040631296)]} 0 4
[junit4:junit4] 2> 28014 T49 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:41267/gi_/collection1/]
[junit4:junit4] 2> 28014 T49 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:41267/gi_/collection1/ against:[http://127.0.0.1:41267/gi_/collection1/] result:true
[junit4:junit4] 2> 28015 T49 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35494/gi_/collection1/ against:[http://127.0.0.1:41267/gi_/collection1/] result:false
[junit4:junit4] 2> 28015 T49 C9 P39577 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45480/gi_/collection1/ against:[http://127.0.0.1:41267/gi_/collection1/] result:false
[junit4:junit4] 2> 28025 T82 C11 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[3 (1438552794054262784)]} 0 4
[junit4:junit4] 2> 28025 T100 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:39577/gi_/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[3 (1438552794054262784)]} 0 4
[junit4:junit4] 2> 28027 T49 C9 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={distrib.from=http://127.0.0.1:45480/gi_/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&wt=javabin&version=2} {add=[3 (1438552794054262784)]} 0 14
[junit4:junit4] 2> 28027 T101 C10 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={test.distrib.skip.servers=http://127.0.0.1:41267/gi_/collection1/&test.distrib.skip.servers=http://127.0.0.1:35494/gi_/collection1/&wt=javabin&version=2} {add=[3]} 0 17
[junit4:junit4] 2> 28031 T28 C13 P45254 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C14 name=collection1 org.apache.solr.core.SolrCore@4ca96e78 url=http://127.0.0.1:45254/gi_/collection1 node=127.0.0.1:45254_gi_ C14_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:45254_gi_, base_url=http://127.0.0.1:45254/gi_, leader=true}
[junit4:junit4] 2> 28136 T28 C14 P45254 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 28136 T28 C14 P45254 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 28164 T28 C14 P45254 oass.SolrIndexSearcher.<init> Opening Searcher@72215167 main
[junit4:junit4] 2> 28165 T28 C14 P45254 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 28167 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72215167 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 28168 T28 C14 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 137
[junit4:junit4] 2> ASYNC NEW_CORE C15 name=collection1 org.apache.solr.core.SolrCore@c1e18f2 url=http://127.0.0.1:45480/gi_/collection1 node=127.0.0.1:45480_gi_ C15_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:45480_gi_, base_url=http://127.0.0.1:45480/gi_}
[junit4:junit4] 2> 28172 T99 C15 P45480 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39577/gi_/collection1/, StdNode: http://127.0.0.1:41267/gi_/collection1/, StdNode: http://127.0.0.1:35494/gi_/collection1/, StdNode: http://127.0.0.1:45480/gi_/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C16 name=collection1 org.apache.solr.core.SolrCore@15cb79b7 url=http://127.0.0.1:41267/gi_/collection1 node=127.0.0.1:41267_gi_ C16_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41267_gi_, base_url=http://127.0.0.1:41267/gi_}
[junit4:junit4] 2> 28177 T64 C16 P41267 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28177 T100 C15 P45480 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C17 name=collection1 org.apache.solr.core.SolrCore@61fa0030 url=http://127.0.0.1:35494/gi_/collection1 node=127.0.0.1:35494_gi_ C17_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35494_gi_, base_url=http://127.0.0.1:35494/gi_}
[junit4:junit4] 2> 28177 T81 C17 P35494 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C18 name=collection1 org.apache.solr.core.SolrCore@611efcc2 url=http://127.0.0.1:39577/gi_/collection1 node=127.0.0.1:39577_gi_ C18_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:39577_gi_, base_url=http://127.0.0.1:39577/gi_, leader=true}
[junit4:junit4] 2> 28180 T51 C18 P39577 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28246 T64 C16 P41267 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@211c9201; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 28246 T64 C16 P41267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 28246 T51 C18 P39577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@68069785; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 28246 T81 C17 P35494 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 28248 T81 C17 P35494 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 28250 T64 C16 P41267 oass.SolrIndexSearcher.<init> Opening Searcher@4a862de1 main
[junit4:junit4] 2> 28251 T64 C16 P41267 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 28246 T100 C15 P45480 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@346633b5; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 28251 T81 C17 P35494 oass.SolrIndexSearcher.<init> Opening Searcher@43df8dc0 main
[junit4:junit4] 2> 28253 T73 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a862de1 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 28254 T81 C17 P35494 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 28247 T51 C18 P39577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 28252 T100 C15 P45480 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 28255 T64 C16 P41267 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 79
[junit4:junit4] 2> 28256 T90 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43df8dc0 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 28259 T51 C18 P39577 oass.SolrIndexSearcher.<init> Opening Searcher@7adf485b main
[junit4:junit4] 2> 28260 T81 C17 P35494 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 84
[junit4:junit4] 2> 28259 T100 C15 P45480 oass.SolrIndexSearcher.<init> Opening Searcher@63177167 main
[junit4:junit4] 2> 28261 T100 C15 P45480 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 28262 T57 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7adf485b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 28263 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63177167 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 28265 T100 C15 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 88
[junit4:junit4] 2> 28265 T51 C18 P39577 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 28267 T51 C18 P39577 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 87
[junit4:junit4] 2> 28268 T99 C15 P45480 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 95
[junit4:junit4] 2> 28268 T11 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 28270 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 28271 T11 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 28276 T50 C18 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=2
[junit4:junit4] 2> 28280 T65 C16 P41267 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 28284 T83 C17 P35494 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 28289 T98 C15 P45480 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 28292 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 28302 T84 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 28303 T84 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 28312 T49 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 28313 T49 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 28313 T49 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ START replicas=[http://127.0.0.1:41267/gi_/collection1/, http://127.0.0.1:35494/gi_/collection1/, http://127.0.0.1:45480/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 28328 T66 C16 P41267 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=11
[junit4:junit4] 2> 28330 T100 C15 P45480 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=10
[junit4:junit4] 2> 28335 T49 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ Received 5 versions from 127.0.0.1:45480/gi_/collection1/
[junit4:junit4] 2> 28335 T49 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ Our versions are newer. ourLowThreshold=1438552793868664832 otherHigh=1438552794020708352
[junit4:junit4] 2> 28336 T49 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ Received 3 versions from 127.0.0.1:41267/gi_/collection1/
[junit4:junit4] 2> 28337 T49 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ Our versions are newer. ourLowThreshold=1438552793868664832 otherHigh=1438552793985056768
[junit4:junit4] 2> 28339 T82 C17 P35494 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 28341 T49 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ Received 5 versions from 127.0.0.1:35494/gi_/collection1/
[junit4:junit4] 2> 28341 T49 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ Our versions are newer. ourLowThreshold=1438552793868664832 otherHigh=1438552794020708352
[junit4:junit4] 2> 28342 T49 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39577/gi_ DONE. sync succeeded
[junit4:junit4] 2> 28342 T49 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 28343 T49 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/: try and ask http://127.0.0.1:41267/gi_/collection1/ to sync
[junit4:junit4] 2> 28344 T49 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/: try and ask http://127.0.0.1:35494/gi_/collection1/ to sync
[junit4:junit4] 2> 28345 T49 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/: try and ask http://127.0.0.1:45480/gi_/collection1/ to sync
[junit4:junit4] 2> 28349 T67 C16 P41267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41267/gi_ START replicas=[http://127.0.0.1:39577/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 28352 T98 C15 P45480 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45480/gi_ START replicas=[http://127.0.0.1:39577/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 28352 T83 C17 P35494 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35494/gi_ START replicas=[http://127.0.0.1:39577/gi_/collection1/] nUpdates=100
[junit4:junit4] 2> 28354 T51 C18 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 28355 T67 C16 P41267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:41267/gi_ Received 5 versions from 127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 28357 T67 C16 P41267 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:41267/gi_ Requesting updates from 127.0.0.1:39577/gi_/collection1/n=2 versions=[1438552794054262784, 1438552794020708352]
[junit4:junit4] 2> 28359 T51 C18 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 28360 T48 C18 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 28361 T83 C17 P35494 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35494/gi_ Received 5 versions from 127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 28362 T83 C17 P35494 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35494/gi_ Our versions are newer. ourLowThreshold=1438552793868664832 otherHigh=1438552794020708352
[junit4:junit4] 2> 28362 T83 C17 P35494 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35494/gi_ DONE. sync succeeded
[junit4:junit4] 2> 28363 T83 C17 P35494 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={sync=http://127.0.0.1:39577/gi_/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=12
[junit4:junit4] 2> 28364 T98 C15 P45480 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45480/gi_ Received 5 versions from 127.0.0.1:39577/gi_/collection1/
[junit4:junit4] 2> 28362 T51 C18 P39577 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={distrib=false&getUpdates=1438552794054262784,1438552794020708352&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 28364 T98 C15 P45480 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45480/gi_ Our versions are newer. ourLowThreshold=1438552793868664832 otherHigh=1438552794020708352
[junit4:junit4] 2> 28365 T98 C15 P45480 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45480/gi_ DONE. sync succeeded
[junit4:junit4] 2> 28366 T98 C15 P45480 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={sync=http://127.0.0.1:39577/gi_/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=15
[junit4:junit4] 2> 28364 T49 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/: sync completed with http://127.0.0.1:35494/gi_/collection1/
[junit4:junit4] 2> 28373 T49 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/: sync completed with http://127.0.0.1:45480/gi_/collection1/
[junit4:junit4] 2> 28383 T67 C16 P41267 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438552794020708352), 3 (1438552794054262784)]} 0 17
[junit4:junit4] 2> 28384 T67 C16 P41267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41267/gi_ DONE. sync succeeded
[junit4:junit4] 2> 28385 T67 C16 P41267 oasc.SolrCore.execute [collection1] webapp=/gi_ path=/get params={sync=http://127.0.0.1:39577/gi_/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=37
[junit4:junit4] 2> 28386 T49 oasc.SyncStrategy.syncToMe http://127.0.0.1:39577/gi_/collection1/: sync completed with http://127.0.0.1:41267/gi_/collection1/
[junit4:junit4] 2> 28386 T49 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=74
[junit4:junit4] 2> 28387 T84 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=86
[junit4:junit4] 2> 28388 T11 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 28389 T11 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 28390 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 28391 T11 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 28395 T29 C14 P45254 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28446 T29 C14 P45254 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66eb6899; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 28447 T29 C14 P45254 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 28448 T29 C14 P45254 oass.SolrIndexSearcher.<init> Opening Searcher@100421dc main
[junit4:junit4] 2> 28451 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@100421dc main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 28452 T29 C14 P45254 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 28453 T29 C14 P45254 oasup.LogUpdateProcessor.finish [collection1] webapp=/gi_ path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 58
[junit4:junit4] 2> 28457 T50 C18 P39577 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39577/gi_/collection1/, StdNode: http://127.0.0.1:41267/gi_/collection1/, StdNode: http://127.0.0.1:35494/gi_/collection1/, StdNode: http://127.0.0.1:45480/gi_/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 28461 T82 C17 P35494 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28462 T65 C16 P41267 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28462 T48 C18 P39577 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28465 T101 C15 P45480 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 28477 T82 C17 P35494 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371910825340/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a94f97a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2
[...truncated too long message...]
"shard":"shard1",
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:41267_gi_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41267/gi_"},
[junit4:junit4] 1> "127.0.0.1:35494_gi__collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:35494_gi_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:35494/gi_"},
[junit4:junit4] 1> "127.0.0.1:45480_gi__collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:45480_gi_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45480/gi_",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:45254_gi__collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:45254_gi_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45254/gi_",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:45480_gi_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:35494_gi_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:45254_gi_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:39577_gi_ (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89909547888934926-127.0.0.1:39577_gi__collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89909547888934922-127.0.0.1:35494_gi__collection1-n_0000000005 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89909547888934924-127.0.0.1:45480_gi__collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:45480_gi_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45480/gi_"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89909547888934915-127.0.0.1:45254_gi__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:45254_gi_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45254/gi_"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89909547888934915-127.0.0.1:45254_gi_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89909547888934926-127.0.0.1:39577_gi_-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89909547888934924-127.0.0.1:45480_gi_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89909547888934922-127.0.0.1:35494_gi_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89909547888934915-127.0.0.1:45254_gi_-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=9663CFCF20185844 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=da_DK -Dtests.timezone=America/Jamaica -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 110s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:39577/gi_/collection1lastClient and got 265 from http://127.0.0.1:35494/gi_/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([9663CFCF20185844:178541D757473878]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 110658 T11 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 109903 T10 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 110752 T88 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 110753 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 110753 T88 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 111840 T105 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 111841 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 111841 T105 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=da_DK, timezone=America/Jamaica
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=66521656,total=126418944
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SyncSliceTest]
[junit4:junit4] Completed on J0 in 112.09s, 1 test, 1 failure <<< FAILURES!
[...truncated 898 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 37 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_25) - Build #
6258 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6258/
Java: 32bit/jdk1.7.0_25 -client -XX:+UseG1GC
2 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 58 from http://127.0.0.1:42546/g_kkf/i/collection1lastClient and got 37 from http://127.0.0.1:40067/g_kkf/i/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 58 from http://127.0.0.1:42546/g_kkf/i/collection1lastClient and got 37 from http://127.0.0.1:40067/g_kkf/i/collection1
at __randomizedtesting.SeedInfo.seed([6A961F23657BD8EF:EB70913B1224B8D3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:59417/collection1lastClient and got 255 from http://127.0.0.1:33417/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:59417/collection1lastClient and got 255 from http://127.0.0.1:33417/collection1
at __randomizedtesting.SeedInfo.seed([6A961F23657BD8EF:EB70913B1224B8D3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9225 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 126160 T203 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 126164 T203 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371920632063
[junit4:junit4] 2> 126166 T203 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 126168 T204 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 126268 T203 oasc.ZkTestServer.run start zk server on port:49534
[junit4:junit4] 2> 126314 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 126554 T210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ce73bc name:ZooKeeperConnection Watcher:127.0.0.1:49534 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 126555 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 126556 T203 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 126588 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 126590 T212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@162e714 name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 126590 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 126592 T203 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 126596 T203 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 126600 T203 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 126603 T203 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 126606 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 126614 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 126620 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 126621 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 126624 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 126625 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 126630 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 126630 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 126634 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 126634 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 126637 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 126637 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 126641 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 126642 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 126645 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 126646 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 126649 T203 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 126650 T203 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 126866 T203 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 126964 T203 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52442
[junit4:junit4] 2> 126977 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 126978 T203 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 126978 T203 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553
[junit4:junit4] 2> 126978 T203 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/solr.xml
[junit4:junit4] 2> 126978 T203 oasc.CoreContainer.<init> New CoreContainer 12883832
[junit4:junit4] 2> 126979 T203 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/'
[junit4:junit4] 2> 126979 T203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/'
[junit4:junit4] 2> 127054 T203 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 127055 T203 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 127055 T203 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 127055 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 127056 T203 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 127056 T203 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 127056 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 127057 T203 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 127057 T203 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 127069 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 127292 T203 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 127293 T203 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49534/solr
[junit4:junit4] 2> 127300 T203 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 127301 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 127304 T225 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@194b507 name:ZooKeeperConnection Watcher:127.0.0.1:49534 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 127304 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 127308 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 127316 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 127317 T227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@120a84d name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 127317 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 127324 T203 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 127330 T203 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 127339 T203 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 127341 T203 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52442_
[junit4:junit4] 2> 127345 T203 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52442_
[junit4:junit4] 2> 127349 T203 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 127361 T203 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 127365 T203 oasc.Overseer.start Overseer (id=89910190549172227-127.0.0.1:52442_-n_0000000000) starting
[junit4:junit4] 2> 127378 T203 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 127389 T229 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 127390 T203 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 127396 T203 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 127398 T203 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 127431 T228 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 127453 T230 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 127453 T230 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 128942 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 128944 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52442_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52442"}
[junit4:junit4] 2> 128945 T228 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 128959 T228 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 128976 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 129455 T230 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/collection1
[junit4:junit4] 2> 129456 T230 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 129457 T230 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 129458 T230 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 129462 T230 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/collection1/'
[junit4:junit4] 2> 129463 T230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/collection1/lib/README' to classloader
[junit4:junit4] 2> 129464 T230 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 129508 T230 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 129548 T230 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 129550 T230 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 129558 T230 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 129987 T230 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 129994 T230 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 129997 T230 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 130031 T230 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 130036 T230 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 130041 T230 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 130043 T230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 130043 T230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 130043 T230 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 130045 T230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 130045 T230 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 130045 T230 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 130055 T230 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371920632553/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/
[junit4:junit4] 2> 130055 T230 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a02
[junit4:junit4] 2> 130056 T230 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 130058 T230 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data
[junit4:junit4] 2> 130058 T230 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index/
[junit4:junit4] 2> 130058 T230 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 130059 T230 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index
[junit4:junit4] 2> 130063 T230 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 130064 T230 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 130087 T230 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 130087 T230 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 130088 T230 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 130088 T230 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 130089 T230 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 130089 T230 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 130089 T230 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 130089 T230 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 130090 T230 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 130092 T230 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 130095 T230 oass.SolrIndexSearcher.<init> Opening Searcher@1d558b2 main
[junit4:junit4] 2> 130096 T230 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 130096 T230 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 130103 T231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d558b2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 130106 T230 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 130107 T230 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52442 collection:control_collection shard:shard1
[junit4:junit4] 2> 130115 T230 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 130127 T230 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 130141 T230 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 130146 T230 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 130147 T230 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 130150 T230 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52442/collection1/
[junit4:junit4] 2> 130151 T230 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 130152 T230 oasc.SyncStrategy.syncToMe http://127.0.0.1:52442/collection1/ has no replicas
[junit4:junit4] 2> 130152 T230 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52442/collection1/
[junit4:junit4] 2> 130153 T230 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 130485 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 130501 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 130515 T230 oasc.ZkController.register We are http://127.0.0.1:52442/collection1/ and leader is http://127.0.0.1:52442/collection1/
[junit4:junit4] 2> 130516 T230 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52442
[junit4:junit4] 2> 130516 T230 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 130517 T230 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 130517 T230 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 130520 T230 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 130523 T203 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 130523 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 130540 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 130546 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 130549 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130550 T234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62a795 name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130550 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130552 T203 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 130556 T203 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 130635 T203 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 130637 T203 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59417
[junit4:junit4] 2> 130637 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 130638 T203 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 130638 T203 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455
[junit4:junit4] 2> 130638 T203 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/solr.xml
[junit4:junit4] 2> 130639 T203 oasc.CoreContainer.<init> New CoreContainer 747353
[junit4:junit4] 2> 130639 T203 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/'
[junit4:junit4] 2> 130639 T203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/'
[junit4:junit4] 2> 130692 T203 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 130692 T203 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 130693 T203 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 130693 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 130693 T203 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 130694 T203 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 130694 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 130694 T203 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 130695 T203 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 130695 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 130699 T203 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 130699 T203 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49534/solr
[junit4:junit4] 2> 130699 T203 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 130700 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130702 T245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5297ed name:ZooKeeperConnection Watcher:127.0.0.1:49534 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130702 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130703 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 130707 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 130709 T247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104a5bc name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 130709 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 130713 T203 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 131716 T203 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59417_
[junit4:junit4] 2> 131718 T203 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59417_
[junit4:junit4] 2> 131724 T227 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 131724 T247 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 131724 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 131724 T234 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 131737 T248 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 131737 T248 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 132012 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 132014 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52442_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52442"}
[junit4:junit4] 2> 132020 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59417_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59417"}
[junit4:junit4] 2> 132020 T228 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 132021 T228 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 132028 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 132028 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 132028 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 132739 T248 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/collection1
[junit4:junit4] 2> 132740 T248 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 132741 T248 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 132741 T248 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 132742 T248 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/collection1/'
[junit4:junit4] 2> 132743 T248 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/collection1/lib/README' to classloader
[junit4:junit4] 2> 132743 T248 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 132773 T248 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 132812 T248 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 132813 T248 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 132819 T248 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 133229 T248 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 133235 T248 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 133238 T248 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 133251 T248 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 133255 T248 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 133258 T248 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 133259 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 133260 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 133260 T248 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 133261 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 133261 T248 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 133261 T248 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 133262 T248 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371920636455/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/
[junit4:junit4] 2> 133262 T248 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a02
[junit4:junit4] 2> 133262 T248 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 133263 T248 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1
[junit4:junit4] 2> 133263 T248 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index/
[junit4:junit4] 2> 133263 T248 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 133264 T248 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index
[junit4:junit4] 2> 133267 T248 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 133267 T248 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 133269 T248 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 133269 T248 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 133269 T248 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 133270 T248 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 133271 T248 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 133271 T248 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 133271 T248 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 133272 T248 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 133272 T248 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 133275 T248 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 133278 T248 oass.SolrIndexSearcher.<init> Opening Searcher@349d57 main
[junit4:junit4] 2> 133278 T248 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 133278 T248 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 133297 T249 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@349d57 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 133300 T248 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 133300 T248 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59417 collection:collection1 shard:shard1
[junit4:junit4] 2> 133301 T248 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 133308 T248 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 133312 T248 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 133312 T248 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 133313 T248 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 133313 T248 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 133313 T248 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/ has no replicas
[junit4:junit4] 2> 133313 T248 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 133313 T248 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 133537 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 133544 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 133544 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 133544 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 133572 T248 oasc.ZkController.register We are http://127.0.0.1:59417/collection1/ and leader is http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 133573 T248 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59417
[junit4:junit4] 2> 133573 T248 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 133574 T248 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 133574 T248 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 133576 T248 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 133577 T203 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 133578 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 133578 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 133653 T203 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 133655 T203 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46309
[junit4:junit4] 2> 133656 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 133656 T203 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 133656 T203 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483
[junit4:junit4] 2> 133657 T203 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/solr.xml
[junit4:junit4] 2> 133657 T203 oasc.CoreContainer.<init> New CoreContainer 4017232
[junit4:junit4] 2> 133657 T203 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/'
[junit4:junit4] 2> 133658 T203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/'
[junit4:junit4] 2> 133711 T203 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 133712 T203 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 133712 T203 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 133712 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 133712 T203 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 133713 T203 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 133713 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 133713 T203 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 133714 T203 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 133714 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 133717 T203 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 133718 T203 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49534/solr
[junit4:junit4] 2> 133718 T203 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 133719 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 133721 T261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10ad99a name:ZooKeeperConnection Watcher:127.0.0.1:49534 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 133721 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 133723 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 133727 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 133728 T263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b7059 name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 133728 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 133731 T203 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 134735 T203 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46309_
[junit4:junit4] 2> 134737 T203 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46309_
[junit4:junit4] 2> 134742 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 134742 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 134742 T227 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 134743 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 134742 T263 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 134744 T234 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 134744 T247 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 134748 T264 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 134748 T264 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 135050 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 135051 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59417_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59417"}
[junit4:junit4] 2> 135054 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46309_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46309"}
[junit4:junit4] 2> 135054 T228 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 135054 T228 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 135058 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 135058 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 135058 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 135058 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 135750 T264 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/collection1
[junit4:junit4] 2> 135750 T264 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 135752 T264 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 135752 T264 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 135754 T264 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/collection1/'
[junit4:junit4] 2> 135756 T264 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/collection1/lib/README' to classloader
[junit4:junit4] 2> 135756 T264 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 135814 T264 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 135854 T264 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 135856 T264 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 135860 T264 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 136271 T264 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 136277 T264 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 136280 T264 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 136299 T264 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 136306 T264 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 136313 T264 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 136315 T264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 136316 T264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 136316 T264 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 136318 T264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 136318 T264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 136318 T264 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 136319 T264 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371920639483/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/
[junit4:junit4] 2> 136319 T264 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a02
[junit4:junit4] 2> 136320 T264 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 136321 T264 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2
[junit4:junit4] 2> 136321 T264 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index/
[junit4:junit4] 2> 136322 T264 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 136323 T264 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index
[junit4:junit4] 2> 136329 T264 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 136329 T264 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 136334 T264 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 136335 T264 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 136336 T264 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 136338 T264 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 136339 T264 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 136339 T264 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 136340 T264 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 136341 T264 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 136342 T264 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 136347 T264 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 136353 T264 oass.SolrIndexSearcher.<init> Opening Searcher@154945 main
[junit4:junit4] 2> 136355 T264 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 136355 T264 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 136365 T265 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@154945 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 136367 T264 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 136367 T264 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46309 collection:collection1 shard:shard1
[junit4:junit4] 2> 136372 T264 oasc.ZkController.register We are http://127.0.0.1:46309/collection1/ and leader is http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 136372 T264 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46309
[junit4:junit4] 2> 136372 T264 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 136372 T264 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C327 name=collection1 org.apache.solr.core.SolrCore@12691d7 url=http://127.0.0.1:46309/collection1 node=127.0.0.1:46309_ C327_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:46309_, base_url=http://127.0.0.1:46309}
[junit4:junit4] 2> 136375 T266 C327 P46309 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 136376 T264 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 136376 T266 C327 P46309 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 136376 T266 C327 P46309 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 136376 T266 C327 P46309 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 136377 T203 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 136377 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 136377 T266 C327 P46309 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 136378 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 136474 T203 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 136476 T203 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33417
[junit4:junit4] 2> 136476 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 136476 T203 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 136477 T203 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283
[junit4:junit4] 2> 136477 T203 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/solr.xml
[junit4:junit4] 2> 136478 T203 oasc.CoreContainer.<init> New CoreContainer 26494938
[junit4:junit4] 2> 136478 T203 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/'
[junit4:junit4] 2> 136478 T203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/'
[junit4:junit4] 2> 136543 T203 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 136543 T203 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 136544 T203 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 136544 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 136544 T203 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 136545 T203 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 136545 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 136545 T203 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 136546 T203 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 136546 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 136550 T203 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 136551 T203 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49534/solr
[junit4:junit4] 2> 136551 T203 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 136552 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 136553 T278 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d68de9 name:ZooKeeperConnection Watcher:127.0.0.1:49534 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 136554 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 136555 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 136559 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 136560 T280 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed3a38 name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 136560 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 136564 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 136565 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46309_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46309"}
[junit4:junit4] 2> 136567 T203 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 136569 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 136569 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 136569 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 136569 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 136569 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 136617 T242 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 136617 T242 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
[junit4:junit4] 2> 136617 T242 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:46309_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2
[junit4:junit4] 2> 137570 T203 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33417_
[junit4:junit4] 2> 137572 T203 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33417_
[junit4:junit4] 2> 137574 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 137574 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 137574 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 137574 T263 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 137574 T227 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 137575 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 137575 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 137576 T234 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 137576 T247 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 137577 T280 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 137581 T281 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 137581 T281 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 138075 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 138077 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33417_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33417"}
[junit4:junit4] 2> 138077 T228 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 138078 T228 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 138084 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 138084 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 138084 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 138084 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 138085 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 138583 T281 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/collection1
[junit4:junit4] 2> 138583 T281 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 138584 T281 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 138584 T281 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 138585 T281 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/collection1/'
[junit4:junit4] 2> 138586 T281 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/collection1/lib/README' to classloader
[junit4:junit4] 2> 138586 T281 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 138638 T281 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> ASYNC NEW_CORE C328 name=collection1 org.apache.solr.core.SolrCore@12691d7 url=http://127.0.0.1:46309/collection1 node=127.0.0.1:46309_ C328_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:46309_, base_url=http://127.0.0.1:46309}
[junit4:junit4] 2> 138648 T266 C328 P46309 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59417/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 138651 T266 C328 P46309 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 138657 T266 C328 P46309 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46309 START replicas=[http://127.0.0.1:59417/collection1/] nUpdates=100
[junit4:junit4] 2> 138661 T266 C328 P46309 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 138662 T266 C328 P46309 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 138662 T266 C328 P46309 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 138662 T266 C328 P46309 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 138664 T266 C328 P46309 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 138664 T266 C328 P46309 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59417/collection1/. core=collection1
[junit4:junit4] 2> 138665 T266 C328 P46309 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C329 name=collection1 org.apache.solr.core.SolrCore@1f0ac1f url=http://127.0.0.1:59417/collection1 node=127.0.0.1:59417_ C329_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59417_, base_url=http://127.0.0.1:59417, leader=true}
[junit4:junit4] 2> 138696 T242 C329 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 138704 T240 C329 P59417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 138707 T240 C329 P59417 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 138707 T240 C329 P59417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 138708 T281 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 138710 T281 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 138717 T240 C329 P59417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 138717 T240 C329 P59417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 138718 T240 C329 P59417 oass.SolrIndexSearcher.<init> Opening Searcher@e7f13b realtime
[junit4:junit4] 2> 138718 T240 C329 P59417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 138719 T240 C329 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4] 2> 138720 T281 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 138746 T266 C328 P46309 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 138747 T266 C328 P46309 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 138755 T266 C328 P46309 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 138800 T242 C329 P59417 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 138801 T242 C329 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=44
[junit4:junit4] 2> 138804 T266 C328 P46309 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 138805 T266 C328 P46309 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 138805 T266 C328 P46309 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 138808 T240 C329 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 138809 T266 C328 P46309 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 138818 T266 C328 P46309 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index.20130623043404709
[junit4:junit4] 2> 138819 T266 C328 P46309 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index.20130623043404709 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7e0c82; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 138827 T243 C329 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
[junit4:junit4] 2> 138835 T266 C328 P46309 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> ASYNC NEW_CORE C330 name=collection1 org.apache.solr.core.SolrCore@12691d7 url=http://127.0.0.1:46309/collection1 node=127.0.0.1:46309_ C330_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:46309_, base_url=http://127.0.0.1:46309}
[junit4:junit4] 2> 138882 T266 C330 P46309 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 138882 T266 C330 P46309 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 138885 T266 C330 P46309 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 138887 T266 C330 P46309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 138887 T266 C330 P46309 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 138888 T266 C330 P46309 oass.SolrIndexSearcher.<init> Opening Searcher@e07c4 main
[junit4:junit4] 2> 138889 T265 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e07c4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 138890 T266 C330 P46309 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index.20130623043404709 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index.20130623043404709;done=true>>]
[junit4:junit4] 2> 138890 T266 C330 P46309 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index.20130623043404709
[junit4:junit4] 2> 138890 T266 C330 P46309 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index.20130623043404709
[junit4:junit4] 2> 138893 T266 C330 P46309 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 138893 T266 C330 P46309 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 138894 T266 C330 P46309 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 138894 T266 C330 P46309 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 138896 T266 C330 P46309 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 139268 T281 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 139275 T281 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 139278 T281 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 139290 T281 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 139295 T281 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 139299 T281 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 139300 T281 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 139300 T281 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 139300 T281 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 139301 T281 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 139302 T281 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 139302 T281 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 139302 T281 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371920642283/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/
[junit4:junit4] 2> 139302 T281 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a02
[junit4:junit4] 2> 139303 T281 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 139303 T281 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3
[junit4:junit4] 2> 139303 T281 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index/
[junit4:junit4] 2> 139304 T281 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 139304 T281 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index
[junit4:junit4] 2> 139328 T281 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 139329 T281 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 139332 T281 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 139332 T281 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 139333 T281 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 139334 T281 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 139335 T281 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 139336 T281 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 139340 T281 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 139341 T281 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 139341 T281 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 139345 T281 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 139350 T281 oass.SolrIndexSearcher.<init> Opening Searcher@544183 main
[junit4:junit4] 2> 139351 T281 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 139351 T281 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 139362 T284 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@544183 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 139365 T281 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 139366 T281 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33417 collection:collection1 shard:shard1
[junit4:junit4] 2> 139369 T281 oasc.ZkController.register We are http://127.0.0.1:33417/collection1/ and leader is http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 139369 T281 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33417
[junit4:junit4] 2> 139369 T281 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 139370 T281 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C331 name=collection1 org.apache.solr.core.SolrCore@19aec75 url=http://127.0.0.1:33417/collection1 node=127.0.0.1:33417_ C331_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:33417_, base_url=http://127.0.0.1:33417}
[junit4:junit4] 2> 139370 T285 C331 P33417 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 139371 T285 C331 P33417 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 139371 T281 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 139371 T285 C331 P33417 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 139371 T285 C331 P33417 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 139372 T203 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 139372 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 139372 T285 C331 P33417 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 139373 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 139384 T243 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 139451 T203 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 139453 T203 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56608
[junit4:junit4] 2> 139454 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 139454 T203 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 139455 T203 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276
[junit4:junit4] 2> 139455 T203 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/solr.xml
[junit4:junit4] 2> 139455 T203 oasc.CoreContainer.<init> New CoreContainer 19038944
[junit4:junit4] 2> 139456 T203 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/'
[junit4:junit4] 2> 139456 T203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/'
[junit4:junit4] 2> 139506 T203 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 139506 T203 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 139506 T203 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 139507 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 139507 T203 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 139507 T203 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 139507 T203 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 139508 T203 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 139508 T203 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 139508 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 139513 T203 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 139513 T203 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49534/solr
[junit4:junit4] 2> 139514 T203 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 139514 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 139517 T297 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@142b3a5 name:ZooKeeperConnection Watcher:127.0.0.1:49534 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 139517 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 139519 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 139524 T203 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 139525 T299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b52b7 name:ZooKeeperConnection Watcher:127.0.0.1:49534/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 139526 T203 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 139530 T203 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 139591 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 139592 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46309_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46309"}
[junit4:junit4] 2> 139596 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33417_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33417"}
[junit4:junit4] 2> 139600 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 139600 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 139600 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 139600 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 139600 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 139600 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 140385 T243 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 140385 T243 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:33417_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 140533 T203 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56608_
[junit4:junit4] 2> 140536 T203 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56608_
[junit4:junit4] 2> 140537 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 140538 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 140538 T227 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 140538 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 140538 T263 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 140537 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 140539 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 140540 T299 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 140540 T234 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 140540 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 140541 T247 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 140541 T280 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 140548 T300 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 140548 T300 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 141106 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 141106 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56608_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56608"}
[junit4:junit4] 2> 141107 T228 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 141107 T228 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 141109 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 141110 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 141109 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 141110 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 141110 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 141110 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 141550 T300 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/collection1
[junit4:junit4] 2> 141550 T300 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 141551 T300 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 141552 T300 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 141553 T300 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/collection1/'
[junit4:junit4] 2> 141554 T300 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/collection1/lib/README' to classloader
[junit4:junit4] 2> 141555 T300 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 141594 T300 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 141639 T300 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 141641 T300 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 141645 T300 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 142064 T300 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 142071 T300 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 142074 T300 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 142086 T300 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 142089 T300 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 142092 T300 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 142093 T300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 142094 T300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 142094 T300 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 142095 T300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 142095 T300 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 142095 T300 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 142095 T300 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371920645276/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/
[junit4:junit4] 2> 142096 T300 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@33a02
[junit4:junit4] 2> 142096 T300 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 142097 T300 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4
[junit4:junit4] 2> 142097 T300 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index/
[junit4:junit4] 2> 142097 T300 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 142097 T300 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index
[junit4:junit4] 2> 142101 T300 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 142101 T300 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 142104 T300 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 142105 T300 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 142106 T300 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 142108 T300 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 142109 T300 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 142110 T300 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 142110 T300 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 142112 T300 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 142113 T300 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 142118 T300 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 142124 T300 oass.SolrIndexSearcher.<init> Opening Searcher@379284 main
[junit4:junit4] 2> 142125 T300 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 142125 T300 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 142134 T301 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@379284 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 142140 T300 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 142140 T300 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56608 collection:collection1 shard:shard1
[junit4:junit4] 2> 142145 T300 oasc.ZkController.register We are http://127.0.0.1:56608/collection1/ and leader is http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 142145 T300 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56608
[junit4:junit4] 2> 142146 T300 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 142146 T300 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C332 name=collection1 org.apache.solr.core.SolrCore@12486af url=http://127.0.0.1:56608/collection1 node=127.0.0.1:56608_ C332_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56608_, base_url=http://127.0.0.1:56608}
[junit4:junit4] 2> 142147 T302 C332 P56608 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 142148 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 142148 T300 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 142148 T302 C332 P56608 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 142149 T302 C332 P56608 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 142151 T203 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 142151 T203 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 142151 T302 C332 P56608 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 142152 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 142161 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 142163 T243 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 142165 T203 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 142165 T203 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 142166 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C333 name=collection1 org.apache.solr.core.SolrCore@19aec75 url=http://127.0.0.1:33417/collection1 node=127.0.0.1:33417_ C333_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:33417_, base_url=http://127.0.0.1:33417}
[junit4:junit4] 2> 142386 T285 C333 P33417 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59417/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 142387 T285 C333 P33417 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33417 START replicas=[http://127.0.0.1:59417/collection1/] nUpdates=100
[junit4:junit4] 2> 142388 T285 C333 P33417 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 142389 T285 C333 P33417 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 142389 T285 C333 P33417 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 142390 T285 C333 P33417 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 142390 T285 C333 P33417 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 142390 T285 C333 P33417 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59417/collection1/. core=collection1
[junit4:junit4] 2> 142391 T285 C333 P33417 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C334 name=collection1 org.apache.solr.core.SolrCore@1f0ac1f url=http://127.0.0.1:59417/collection1 node=127.0.0.1:59417_ C334_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59417_, base_url=http://127.0.0.1:59417, leader=true}
[junit4:junit4] 2> 142392 T241 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 142400 T240 C334 P59417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 142407 T240 C334 P59417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 142408 T240 C334 P59417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 142409 T240 C334 P59417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 142409 T240 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 142411 T285 C333 P33417 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 142411 T285 C333 P33417 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 142416 T241 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 142417 T285 C333 P33417 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 142417 T285 C333 P33417 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 142418 T285 C333 P33417 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 142421 T240 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=1
[junit4:junit4] 2> 142422 T285 C333 P33417 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 142423 T285 C333 P33417 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index.20130623043408322
[junit4:junit4] 2> 142424 T285 C333 P33417 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index.20130623043408322 lockFactory=org.apache.lucene.store.NativeFSLockFactory@a18ffb; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 142428 T242 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 142431 T285 C333 P33417 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 142433 T285 C333 P33417 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 142433 T285 C333 P33417 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 142435 T285 C333 P33417 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 142435 T285 C333 P33417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 142435 T285 C333 P33417 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 142436 T285 C333 P33417 oass.SolrIndexSearcher.<init> Opening Searcher@144edf6 main
[junit4:junit4] 2> 142437 T284 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@144edf6 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 142437 T285 C333 P33417 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index.20130623043408322 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index.20130623043408322;done=true>>]
[junit4:junit4] 2> 142438 T285 C333 P33417 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index.20130623043408322
[junit4:junit4] 2> 142438 T285 C333 P33417 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index.20130623043408322
[junit4:junit4] 2> 142438 T285 C333 P33417 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 142438 T285 C333 P33417 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 142439 T285 C333 P33417 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 142439 T285 C333 P33417 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 142440 T285 C333 P33417 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 142615 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 142616 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56608_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56608"}
[junit4:junit4] 2> 142640 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33417_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33417"}
[junit4:junit4] 2> 142644 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 142644 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 142644 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 142644 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 142644 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 142644 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 143164 T243 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 143164 T243 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:56608_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 143169 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 144170 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C332_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56608_, base_url=http://127.0.0.1:56608}
[junit4:junit4] 2> 145165 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59417/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 145166 T302 C332 P56608 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56608 START replicas=[http://127.0.0.1:59417/collection1/] nUpdates=100
[junit4:junit4] 2> 145166 T302 C332 P56608 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 145166 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 145166 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 145166 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 145167 T302 C332 P56608 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 145167 T302 C332 P56608 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59417/collection1/. core=collection1
[junit4:junit4] 2> 145167 T302 C332 P56608 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 145168 T241 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 145172 T242 C334 P59417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 145172 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 145177 T242 C334 P59417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 145177 T242 C334 P59417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 145178 T242 C334 P59417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 145178 T242 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 145179 T302 C332 P56608 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 145179 T302 C332 P56608 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 145181 T243 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 145182 T302 C332 P56608 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 145182 T302 C332 P56608 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 145182 T302 C332 P56608 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 145185 T241 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 145185 T302 C332 P56608 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 145186 T302 C332 P56608 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index.20130623043411085
[junit4:junit4] 2> 145186 T302 C332 P56608 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index.20130623043411085 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c8752a; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 145189 T242 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 145191 T302 C332 P56608 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 145193 T302 C332 P56608 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 145193 T302 C332 P56608 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 145195 T302 C332 P56608 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 145195 T302 C332 P56608 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 145195 T302 C332 P56608 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 145196 T302 C332 P56608 oass.SolrIndexSearcher.<init> Opening Searcher@f6ec10 main
[junit4:junit4] 2> 145197 T301 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f6ec10 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 145198 T302 C332 P56608 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index.20130623043411085 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index.20130623043411085;done=true>>]
[junit4:junit4] 2> 145198 T302 C332 P56608 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index.20130623043411085
[junit4:junit4] 2> 145199 T302 C332 P56608 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index.20130623043411085
[junit4:junit4] 2> 145199 T302 C332 P56608 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 145199 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 145199 T302 C332 P56608 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 145200 T302 C332 P56608 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 145201 T302 C332 P56608 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 145651 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 145652 T228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56608_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56608"}
[junit4:junit4] 2> 145655 T227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 145655 T299 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 145655 T263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 145655 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 145655 T234 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 145655 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 146174 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 146176 T203 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C335 name=collection1 org.apache.solr.core.SolrCore@17120df url=http://127.0.0.1:52442/collection1 node=127.0.0.1:52442_ C335_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:52442_, base_url=http://127.0.0.1:52442, leader=true}
[junit4:junit4] 2> 146185 T223 C335 P52442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 146188 T223 C335 P52442 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 146189 T223 C335 P52442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 146197 T223 C335 P52442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 146197 T223 C335 P52442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 146198 T223 C335 P52442 oass.SolrIndexSearcher.<init> Opening Searcher@3bdb3a main
[junit4:junit4] 2> 146198 T223 C335 P52442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 146200 T231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3bdb3a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 146201 T223 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> ASYNC NEW_CORE C336 name=collection1 org.apache.solr.core.SolrCore@12691d7 url=http://127.0.0.1:46309/collection1 node=127.0.0.1:46309_ C336_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:46309_, base_url=http://127.0.0.1:46309}
[junit4:junit4] 2> 146218 T258 C336 P46309 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:59417/collection1/, StdNode: http://127.0.0.1:46309/collection1/, StdNode: http://127.0.0.1:33417/collection1/, StdNode: http://127.0.0.1:56608/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 146226 T240 C334 P59417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 146226 T259 C336 P46309 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C333_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:33417_, base_url=http://127.0.0.1:33417}
[junit4:junit4] 2> 146232 T275 C333 P33417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C332_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56608_, base_url=http://127.0.0.1:56608}
[junit4:junit4] 2> 146232 T295 C332 P56608 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 146257 T259 C336 P46309 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 146258 T259 C336 P46309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 146257 T240 C334 P59417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 146258 T259 C336 P46309 oass.SolrIndexSearcher.<init> Opening Searcher@13e19d8 main
[junit4:junit4] 2> 146258 T240 C334 P59417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 146259 T259 C336 P46309 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 146259 T240 C334 P59417 oass.SolrIndexSearcher.<init> Opening Searcher@693c42 main
[junit4:junit4] 2> 146260 T240 C334 P59417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 146260 T265 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13e19d8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 146261 T259 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 35
[junit4:junit4] 2> 146261 T249 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@693c42 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 146262 T240 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 36
[junit4:junit4] 2> 146277 T295 C332 P56608 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 146278 T295 C332 P56608 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 146277 T275 C333 P33417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 146279 T295 C332 P56608 oass.SolrIndexSearcher.<init> Opening Searcher@9afec7 main
[junit4:junit4] 2> 146279 T275 C333 P33417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 146280 T295 C332 P56608 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 146281 T275 C333 P33417 oass.SolrIndexSearcher.<init> Opening Searcher@119b3e5 main
[junit4:junit4] 2> 146282 T301 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9afec7 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 146282 T275 C333 P33417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 146283 T295 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 51
[junit4:junit4] 2> 146283 T284 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@119b3e5 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 146285 T275 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 53
[junit4:junit4] 2> 146286 T258 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 69
[junit4:junit4] 2> 146288 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 146299 T243 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 146302 T256 C336 P46309 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 146304 T273 C333 P33417 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 146306 T294 C332 P56608 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 148332 T223 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438563071929876480)} 0 9
[junit4:junit4] 2> 148340 T294 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438563071933022208&update.from=http://127.0.0.1:59417/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438563071933022208)} 0 1
[junit4:junit4] 2> 148340 T274 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438563071933022208&update.from=http://127.0.0.1:59417/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438563071933022208)} 0 1
[junit4:junit4] 2> 148345 T257 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438563071933022208&update.from=http://127.0.0.1:59417/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438563071933022208)} 0 1
[junit4:junit4] 2> 148346 T240 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438563071933022208)} 0 12
[junit4:junit4] 2> 148358 T222 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438563071949799424)]} 0 9
[junit4:junit4] 2> 148371 T276 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438563071961333760)]} 0 3
[junit4:junit4] 2> 148371 T259 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438563071961333760)]} 0 3
[junit4:junit4] 2> 148372 T292 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438563071961333760)]} 0 4
[junit4:junit4] 2> 148372 T243 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438563071961333760)]} 0 11
[junit4:junit4] 2> 148377 T220 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438563071977062400)]} 0 1
[junit4:junit4] 2> 148387 T275 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438563071984402432)]} 0 1
[junit4:junit4] 2> 148388 T294 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438563071984402432)]} 0 2
[junit4:junit4] 2> 148390 T258 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438563071984402432)]} 0 1
[junit4:junit4] 2> 148391 T242 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:56608/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438563071984402432)]} 0 8
[junit4:junit4] 2> 148391 T293 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 11
[junit4:junit4] 2> 148395 T221 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438563071995936768)]} 0 1
[junit4:junit4] 2> 148399 T240 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:46309/collection1/]
[junit4:junit4] 2> 148399 T240 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:46309/collection1/ against:[http://127.0.0.1:46309/collection1/] result:true
[junit4:junit4] 2> 148400 T240 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:33417/collection1/ against:[http://127.0.0.1:46309/collection1/] result:false
[junit4:junit4] 2> 148400 T240 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:56608/collection1/ against:[http://127.0.0.1:46309/collection1/] result:false
[junit4:junit4] 2> 148405 T292 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[2 (1438563072002228224)]} 0 2
[junit4:junit4] 2> 148405 T274 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[2 (1438563072002228224)]} 0 2
[junit4:junit4] 2> 148406 T240 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33417/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[2 (1438563072002228224)]} 0 7
[junit4:junit4] 2> 148407 T273 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[2]} 0 10
[junit4:junit4] 2> 148410 T223 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438563072011665408)]} 0 1
[junit4:junit4] 2> 148416 T243 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:46309/collection1/]
[junit4:junit4] 2> 148416 T243 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:46309/collection1/ against:[http://127.0.0.1:46309/collection1/] result:true
[junit4:junit4] 2> 148416 T243 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:33417/collection1/ against:[http://127.0.0.1:46309/collection1/] result:false
[junit4:junit4] 2> 148417 T243 C334 P59417 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:56608/collection1/ against:[http://127.0.0.1:46309/collection1/] result:false
[junit4:junit4] 2> 148423 T295 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[3 (1438563072020054016)]} 0 2
[junit4:junit4] 2> 148423 T276 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59417/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[3 (1438563072020054016)]} 0 1
[junit4:junit4] 2> 148424 T243 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:46309/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&wt=javabin&version=2} {add=[3 (1438563072020054016)]} 0 8
[junit4:junit4] 2> 148424 T256 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:46309/collection1/&test.distrib.skip.servers=http://127.0.0.1:33417/collection1/&wt=javabin&version=2} {add=[3]} 0 11
[junit4:junit4] 2> 148427 T222 C335 P52442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148447 T222 C335 P52442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 148447 T222 C335 P52442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 148468 T222 C335 P52442 oass.SolrIndexSearcher.<init> Opening Searcher@676697 main
[junit4:junit4] 2> 148469 T222 C335 P52442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148470 T231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@676697 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 148470 T222 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 43
[junit4:junit4] 2> 148473 T241 C334 P59417 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:59417/collection1/, StdNode: http://127.0.0.1:46309/collection1/, StdNode: http://127.0.0.1:33417/collection1/, StdNode: http://127.0.0.1:56608/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 148476 T242 C334 P59417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148476 T257 C336 P46309 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148477 T294 C332 P56608 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148476 T275 C333 P33417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148527 T257 C336 P46309 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1bd210b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 148528 T257 C336 P46309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 148527 T294 C332 P56608 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 148527 T275 C333 P33417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d31aac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 148529 T275 C333 P33417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 148527 T242 C334 P59417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ceb801; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 148530 T242 C334 P59417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 148528 T294 C332 P56608 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 148530 T257 C336 P46309 oass.SolrIndexSearcher.<init> Opening Searcher@109e5b3 main
[junit4:junit4] 2> 148531 T257 C336 P46309 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148532 T275 C333 P33417 oass.SolrIndexSearcher.<init> Opening Searcher@20ef11 main
[junit4:junit4] 2> 148532 T275 C333 P33417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148533 T265 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@109e5b3 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 148533 T294 C332 P56608 oass.SolrIndexSearcher.<init> Opening Searcher@53b59b main
[junit4:junit4] 2> 148534 T294 C332 P56608 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148534 T257 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 58
[junit4:junit4] 2> 148534 T284 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20ef11 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 148535 T275 C333 P33417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 59
[junit4:junit4] 2> 148535 T301 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@53b59b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 148535 T242 C334 P59417 oass.SolrIndexSearcher.<init> Opening Searcher@c1996d main
[junit4:junit4] 2> 148536 T242 C334 P59417 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148536 T294 C332 P56608 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 59
[junit4:junit4] 2> 148538 T249 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c1996d main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 148539 T242 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 63
[junit4:junit4] 2> 148539 T241 C334 P59417 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 66
[junit4:junit4] 2> 148540 T203 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 148541 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 148542 T203 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 148545 T240 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 148548 T259 C336 P46309 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 148550 T274 C333 P33417 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 148553 T293 C332 P56608 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 148554 T203 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 148560 T273 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 148560 T273 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 148564 T243 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 148565 T243 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59417/collection1/
[junit4:junit4] 2> 148565 T243 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59417 START replicas=[http://127.0.0.1:46309/collection1/, http://127.0.0.1:33417/collection1/, http://127.0.0.1:56608/collection1/] nUpdates=100
[junit4:junit4] 2> 148573 T295 C332 P56608 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 148574 T256 C336 P46309 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 148574 T275 C333 P33417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 148574 T243 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59417 Received 5 versions from 127.0.0.1:56608/collection1/
[junit4:junit4] 2> 148575 T243 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59417 Our versions are newer. ourLowThreshold=1438563071933022208 otherHigh=1438563072002228224
[junit4:junit4] 2> 148575 T243 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59417 Received 3 versions from 127.0.0.1:46309/collection1/
[junit4:junit4] 2> 148575 T243 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59417 Our versions are newer. ourLowThreshold=1438563071933022208 otherHigh=1438563071984402432
[junit4:junit4] 2> 148576 T243 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59417 Received 5 versions from 127.0.0.1:33417/collection1/
[junit4:junit4] 2> 148576 T243 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59417 Our versions are newer. ourLowThreshold=1438563071933022208 otherHigh=1438563072002228224
[junit4:junit4] 2> 148576 T243 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59417 DONE. sync succeeded
[junit4:junit4] 2> 148576 T243 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 148577 T243 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/: try and ask http://127.0.0.1:46309/collection1/ to sync
[junit4:junit4] 2> 148577 T243 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/: try and ask http://127.0.0.1:33417/collection1/ to sync
[junit4:junit4] 2> 148578 T243 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/: try and ask http://127.0.0.1:56608/collection1/ to sync
[junit4:junit4] 2> 148579 T276 C333 P33417 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33417 START replicas=[http://127.0.0.1:59417/collection1/] nUpdates=100
[junit4:junit4] 2> 148579 T257 C336 P46309 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46309 START replicas=[http://127.0.0.1:59417/collection1/] nUpdates=100
[junit4:junit4] 2> 148579 T294 C332 P56608 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56608 START replicas=[http://127.0.0.1:59417/collection1/] nUpdates=100
[junit4:junit4] 2> 148581 T242 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 148582 T240 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 148582 T257 C336 P46309 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46309 Received 5 versions from 127.0.0.1:59417/collection1/
[junit4:junit4] 2> 148582 T241 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 148583 T257 C336 P46309 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:46309 Requesting updates from 127.0.0.1:59417/collection1/n=2 versions=[1438563072020054016, 1438563072002228224]
[junit4:junit4] 2> 148583 T294 C332 P56608 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56608 Received 5 versions from 127.0.0.1:59417/collection1/
[junit4:junit4] 2> 148582 T276 C333 P33417 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33417 Received 5 versions from 127.0.0.1:59417/collection1/
[junit4:junit4] 2> 148584 T294 C332 P56608 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56608 Our versions are newer. ourLowThreshold=1438563071933022208 otherHigh=1438563072002228224
[junit4:junit4] 2> 148584 T276 C333 P33417 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33417 Our versions are newer. ourLowThreshold=1438563071933022208 otherHigh=1438563072002228224
[junit4:junit4] 2> 148584 T294 C332 P56608 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56608 DONE. sync succeeded
[junit4:junit4] 2> 148585 T276 C333 P33417 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33417 DONE. sync succeeded
[junit4:junit4] 2> 148585 T276 C333 P33417 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:59417/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 148585 T294 C332 P56608 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:59417/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 148586 T243 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/: sync completed with http://127.0.0.1:33417/collection1/
[junit4:junit4] 2> 148586 T242 C334 P59417 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1438563072020054016,1438563072002228224&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 148586 T243 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/: sync completed with http://127.0.0.1:56608/collection1/
[junit4:junit4] 2> 148593 T257 C336 P46309 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438563072002228224), 3 (1438563072020054016)]} 0 3
[junit4:junit4] 2> 148593 T257 C336 P46309 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46309 DONE. sync succeeded
[junit4:junit4] 2> 148593 T257 C336 P46309 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:59417/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=14
[junit4:junit4] 2> 148594 T243 oasc.SyncStrategy.syncToMe http://127.0.0.1:59417/collection1/: sync completed with http://127.0.0.1:46309/collection1/
[junit4:junit4] 2> 148594 T243 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=30
[junit4:junit4] 2> 148595 T273 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=36
[junit4:junit4] 2> 148595 T203 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 148596 T203 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 148596 T203 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 148597 T203 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 148599 T221 C335 P52442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148647 T221 C335 P52442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fe4b58; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 148648 T221 C335 P52442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 148648 T221 C335 P52442 oass.SolrIndexSearcher.<init> Opening Searcher@22d26f main
[junit4:junit4] 2> 148649 T221 C335 P52442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 148650 T231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22d26f main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 148650 T221 C335 P52442 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 51
[junit4:junit4] 2> 148653 T275 C333 P33417 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:59417/collection1/, StdNode: http://127.0.0.1:46309/collection1/, StdNode: http://127.0.0.1:33417/collection1/, StdNode: http://127.0.0.1:56608/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 148656 T293 C332 P56608 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148657 T274 C333 P33417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148656 T259 C336 P46309 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148656 T240 C334 P59417 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 148697 T293 C332 P56608 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371920632063/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@e4b088; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_7,generation=7}
[junit4:junit4] 2> 148697 T240 C334 P59417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits
[...truncated too long message...]
junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"11",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:38072_g_kkf%2Fi",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38072/g_kkf/i"}
[junit4:junit4] 2> 572398 T2074 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"11",
[junit4:junit4] 2> "numShards":null,
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38072_g_kkf%2Fi",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38072/g_kkf/i"}
[junit4:junit4] 2> 572403 T2074 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"13",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49996_g_kkf%2Fi",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49996/g_kkf/i"}
[junit4:junit4] 2> 572408 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 573019 T1660 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 573020 T1660 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 573022 T1660 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@16a46bb
[junit4:junit4] 2> 573023 T1660 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=107,cumulative_deletesById=56,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 573025 T1660 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 573026 T1660 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 573026 T1660 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 573028 T1660 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 573029 T1660 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 573030 T1660 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371920980516/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371920980516/jetty12;done=false>>]
[junit4:junit4] 2> 573030 T1660 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371920980516/jetty12
[junit4:junit4] 2> 573031 T1660 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371920980516/jetty12/index.20130622181026916 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371920980516/jetty12/index.20130622181026916;done=false>>]
[junit4:junit4] 2> 573032 T1660 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371920980516/jetty12/index.20130622181026916
[junit4:junit4] 2> 573033 T2074 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89910213379293212-127.0.0.1:49996_g_kkf%2Fi-n_0000000012) am no longer a leader.
[junit4:junit4] 2> 573035 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 573035 T1898 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 573036 T1898 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 573059 T1660 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/g_kkf/i,null}
[junit4:junit4] 2> 573161 T1660 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 573162 T1660 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51625 51625
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=6A961F23657BD8EF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_MA -Dtests.timezone=Africa/Porto-Novo -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 98.6s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard2 is not consistent. Got 58 from http://127.0.0.1:42546/g_kkf/i/collection1lastClient and got 37 from http://127.0.0.1:40067/g_kkf/i/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([6A961F23657BD8EF:EB70913B1224B8D3]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 573181 T1660 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 98643 T1659 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=ar_MA, timezone=Africa/Porto-Novo
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=4,free=287677992,total=536870912
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSchemaNameResource, IndexReaderFactoryTest, TestComponentsName, SolrIndexConfigTest, FileUtilsTest, DebugComponentTest, DefaultValueUpdateProcessorTest, TestLMJelinekMercerSimilarityFactory, TestStressLucene, TestRecovery, OpenCloseCoreStressTest, RequiredFieldsTest, TestAnalyzedSuggestions, TestFieldResource, TermVectorComponentDistributedTest, SampleTest, MultiTermTest, BasicDistributedZk2Test, PathHierarchyTokenizerFactoryTest, TestSchemaVersionResource, OutputWriterTest, SolrRequestParserTest, TestDefaultSearchFieldResource, TestQueryTypes, TestMaxScoreQueryParser, TestArbitraryIndexDir, TestSolrQueryParserResource, HardAutoCommitTest, DateMathParserTest, BasicZkTest, SolrPluginUtilsTest, TestElisionMultitermQuery, TestSystemIdResolver, TestFastOutputStream, FieldAnalysisRequestHandlerTest, TestSolrXMLSerializer, TestMergePolicyConfig, TestCodecSupport, TestZkChroot, DocumentAnalysisRequestHandlerTest, RequestHandlersTest, TestCoreDiscovery, TestCollationField, StandardRequestHandlerTest, CurrencyFieldXmlFileTest, TestUpdate, TestLazyCores, DisMaxRequestHandlerTest, CopyFieldTest, TestCSVLoader, TestDynamicFieldResource, TimeZoneUtilsTest, ShardRoutingCustomTest, TestRTGBase, SimplePostToolTest, BadIndexSchemaTest, TestStressReorder, CollectionsAPIDistributedZkTest, TestPartialUpdateDeduplication, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 99.68s, 1 test, 1 failure <<< FAILURES!
[...truncated 532 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 2 failures, 13 ignored (7 assumptions)
Total time: 41 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b93) - Build #
6257 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6257/
Java: 64bit/jdk1.8.0-ea-b93 -XX:-UseCompressedOops -XX:+UseSerialGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 25 from http://127.0.0.1:54619/vzz/ut/collection1lastClient and got 23 from http://127.0.0.1:47836/vzz/ut/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 25 from http://127.0.0.1:54619/vzz/ut/collection1lastClient and got 23 from http://127.0.0.1:47836/vzz/ut/collection1
at __randomizedtesting.SeedInfo.seed([E4C9D5F38A804DC7:652F5BEBFDDF2DFB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:44408/_cjv/x/collection1lastClient and got 241 from http://127.0.0.1:60443/_cjv/x/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:44408/_cjv/x/collection1lastClient and got 241 from http://127.0.0.1:60443/_cjv/x/collection1
at __randomizedtesting.SeedInfo.seed([E4C9D5F38A804DC7:652F5BEBFDDF2DFB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9440 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 257325 T1016 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /vzz/ut
[junit4:junit4] 2> 257331 T1016 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371916089794
[junit4:junit4] 2> 257332 T1016 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 257332 T1017 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 257432 T1016 oasc.ZkTestServer.run start zk server on port:35160
[junit4:junit4] 2> 257433 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 257610 T1023 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a6ef8a9 name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 257610 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 257610 T1016 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 257616 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 257618 T1025 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74a90994 name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 257618 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 257618 T1016 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 257622 T1016 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 257624 T1016 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 257626 T1016 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 257628 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 257629 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 257633 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 257633 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 257637 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 257637 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 257640 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 257641 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 257644 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 257644 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 257648 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 257648 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 257651 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 257651 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 257654 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 257655 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 257657 T1016 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 257657 T1016 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 257739 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 257764 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44924
[junit4:junit4] 2> 257765 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 257765 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 257765 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123
[junit4:junit4] 2> 257766 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/solr.xml
[junit4:junit4] 2> 257766 T1016 oasc.CoreContainer.<init> New CoreContainer 1529048714
[junit4:junit4] 2> 257766 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/'
[junit4:junit4] 2> 257766 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/'
[junit4:junit4] 2> 257820 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 257821 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 257821 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 257821 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 257821 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 257822 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 257822 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 257822 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 257822 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 257823 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 257827 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 257828 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 257828 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 257829 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 257831 T1036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@895812a name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 257832 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 257834 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 257837 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 257839 T1038 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74451439 name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 257840 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 257841 T1016 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 257845 T1016 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 257848 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 257850 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44924_vzz%2Fut
[junit4:junit4] 2> 257850 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44924_vzz%2Fut
[junit4:junit4] 2> 257853 T1016 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 257857 T1016 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 257859 T1016 oasc.Overseer.start Overseer (id=89909892860870659-127.0.0.1:44924_vzz%2Fut-n_0000000000) starting
[junit4:junit4] 2> 257863 T1016 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 257867 T1040 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 257867 T1016 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 257870 T1016 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 257872 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 257875 T1039 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 257877 T1041 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 257878 T1041 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 259379 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 259379 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44924/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44924_vzz%2Fut"}
[junit4:junit4] 2> 259380 T1039 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 259380 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 259383 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 259879 T1041 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/collection1
[junit4:junit4] 2> 259880 T1041 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 259880 T1041 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 259881 T1041 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 259882 T1041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/collection1/'
[junit4:junit4] 2> 259883 T1041 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/collection1/lib/README' to classloader
[junit4:junit4] 2> 259883 T1041 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 259909 T1041 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 259946 T1041 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 259948 T1041 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 259951 T1041 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 260266 T1041 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 260270 T1041 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 260272 T1041 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 260283 T1041 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 260286 T1041 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 260288 T1041 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 260289 T1041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 260289 T1041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 260290 T1041 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 260290 T1041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 260291 T1041 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 260291 T1041 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 260297 T1041 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371916090123/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/control/data/
[junit4:junit4] 2> 260297 T1041 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 260297 T1041 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 260298 T1041 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/control/data
[junit4:junit4] 2> 260298 T1041 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/control/data/index/
[junit4:junit4] 2> 260298 T1041 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 260298 T1041 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/control/data/index
[junit4:junit4] 2> 260301 T1041 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 260301 T1041 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 260303 T1041 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 260303 T1041 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 260304 T1041 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 260305 T1041 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 260305 T1041 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 260305 T1041 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 260306 T1041 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 260306 T1041 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 260306 T1041 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 260309 T1041 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 260313 T1041 oass.SolrIndexSearcher.<init> Opening Searcher@3d746eb8 main
[junit4:junit4] 2> 260313 T1041 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 260314 T1041 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 260317 T1042 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d746eb8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 260317 T1041 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 260317 T1041 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44924/vzz/ut collection:control_collection shard:shard1
[junit4:junit4] 2> 260318 T1041 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 260323 T1041 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 260325 T1041 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 260325 T1041 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 260325 T1041 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44924/vzz/ut/collection1/
[junit4:junit4] 2> 260326 T1041 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 260326 T1041 oasc.SyncStrategy.syncToMe http://127.0.0.1:44924/vzz/ut/collection1/ has no replicas
[junit4:junit4] 2> 260326 T1041 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44924/vzz/ut/collection1/
[junit4:junit4] 2> 260326 T1041 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 260887 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 260894 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 260933 T1041 oasc.ZkController.register We are http://127.0.0.1:44924/vzz/ut/collection1/ and leader is http://127.0.0.1:44924/vzz/ut/collection1/
[junit4:junit4] 2> 260934 T1041 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44924/vzz/ut
[junit4:junit4] 2> 260934 T1041 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 260934 T1041 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 260934 T1041 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 260936 T1041 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 260937 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 260937 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 260938 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 260941 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 260942 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 260944 T1045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b53f4bc name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 260945 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 260946 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 260948 T1016 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 261021 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 261023 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55690
[junit4:junit4] 2> 261023 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 261024 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 261024 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410
[junit4:junit4] 2> 261024 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/solr.xml
[junit4:junit4] 2> 261024 T1016 oasc.CoreContainer.<init> New CoreContainer 1829338893
[junit4:junit4] 2> 261025 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/'
[junit4:junit4] 2> 261025 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/'
[junit4:junit4] 2> 261065 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 261065 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 261065 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 261066 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 261066 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 261066 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 261066 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 261066 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 261067 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 261067 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 261071 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 261071 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 261071 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 261072 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 261074 T1056 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@646d8e0f name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 261074 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 261076 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 261081 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 261083 T1058 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a028dd5 name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 261083 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 261086 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 262090 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55690_vzz%2Fut
[junit4:junit4] 2> 262091 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55690_vzz%2Fut
[junit4:junit4] 2> 262094 T1038 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 262094 T1058 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 262094 T1045 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 262094 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 262099 T1059 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 262099 T1059 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 262400 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 262401 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44924/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44924_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:44924_vzz%2Fut"}
[junit4:junit4] 2> 262405 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55690/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55690_vzz%2Fut"}
[junit4:junit4] 2> 262405 T1039 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 262405 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 262412 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 262412 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 262412 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 263101 T1059 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/collection1
[junit4:junit4] 2> 263101 T1059 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 263102 T1059 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 263102 T1059 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 263103 T1059 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/collection1/'
[junit4:junit4] 2> 263103 T1059 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/collection1/lib/README' to classloader
[junit4:junit4] 2> 263104 T1059 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 263135 T1059 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 263162 T1059 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 263164 T1059 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 263167 T1059 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 263482 T1059 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 263487 T1059 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 263489 T1059 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 263502 T1059 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 263505 T1059 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 263508 T1059 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 263509 T1059 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 263509 T1059 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 263510 T1059 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 263511 T1059 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 263511 T1059 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 263511 T1059 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 263511 T1059 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371916093410/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/
[junit4:junit4] 2> 263511 T1059 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 263512 T1059 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 263512 T1059 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1
[junit4:junit4] 2> 263512 T1059 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index/
[junit4:junit4] 2> 263513 T1059 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 263513 T1059 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index
[junit4:junit4] 2> 263516 T1059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 263516 T1059 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 263517 T1059 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 263518 T1059 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 263518 T1059 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 263518 T1059 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 263519 T1059 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 263519 T1059 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 263519 T1059 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 263519 T1059 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 263520 T1059 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 263521 T1059 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 263524 T1059 oass.SolrIndexSearcher.<init> Opening Searcher@28bd02e8 main
[junit4:junit4] 2> 263524 T1059 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 263524 T1059 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 263528 T1060 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28bd02e8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 263529 T1059 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 263529 T1059 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55690/vzz/ut collection:collection1 shard:shard3
[junit4:junit4] 2> 263530 T1059 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 263536 T1059 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 263538 T1059 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 263538 T1059 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 263538 T1059 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55690/vzz/ut/collection1/
[junit4:junit4] 2> 263538 T1059 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 263538 T1059 oasc.SyncStrategy.syncToMe http://127.0.0.1:55690/vzz/ut/collection1/ has no replicas
[junit4:junit4] 2> 263538 T1059 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55690/vzz/ut/collection1/
[junit4:junit4] 2> 263539 T1059 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 263918 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 263926 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 263927 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 263927 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 263945 T1059 oasc.ZkController.register We are http://127.0.0.1:55690/vzz/ut/collection1/ and leader is http://127.0.0.1:55690/vzz/ut/collection1/
[junit4:junit4] 2> 263945 T1059 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55690/vzz/ut
[junit4:junit4] 2> 263945 T1059 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 263945 T1059 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 263945 T1059 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 263947 T1059 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 263948 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 263949 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 263949 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 264047 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 264049 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46291
[junit4:junit4] 2> 264050 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 264051 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 264051 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415
[junit4:junit4] 2> 264051 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/solr.xml
[junit4:junit4] 2> 264051 T1016 oasc.CoreContainer.<init> New CoreContainer 2097014703
[junit4:junit4] 2> 264052 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/'
[junit4:junit4] 2> 264052 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/'
[junit4:junit4] 2> 264100 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 264100 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 264100 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 264100 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 264101 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 264101 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 264101 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 264101 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 264102 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 264102 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 264105 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 264106 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 264106 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 264107 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 264110 T1072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7148d1bc name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 264110 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 264112 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 264116 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 264118 T1074 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75423bc0 name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 264118 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 264122 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 265125 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46291_vzz%2Fut
[junit4:junit4] 2> 265126 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46291_vzz%2Fut
[junit4:junit4] 2> 265128 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 265129 T1038 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 265130 T1058 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 265130 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 265130 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 265130 T1074 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 265132 T1045 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 265135 T1075 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 265135 T1075 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 265433 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 265434 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55690/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:55690_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55690_vzz%2Fut"}
[junit4:junit4] 2> 265438 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46291/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46291_vzz%2Fut"}
[junit4:junit4] 2> 265438 T1039 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 265438 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 265443 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 265443 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 265444 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 265443 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 266136 T1075 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/collection1
[junit4:junit4] 2> 266137 T1075 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 266137 T1075 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 266138 T1075 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 266139 T1075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/collection1/'
[junit4:junit4] 2> 266139 T1075 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/collection1/lib/README' to classloader
[junit4:junit4] 2> 266139 T1075 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 266162 T1075 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 266196 T1075 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 266198 T1075 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 266201 T1075 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 266509 T1075 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 266513 T1075 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 266515 T1075 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 266528 T1075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 266531 T1075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 266533 T1075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 266534 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 266535 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 266535 T1075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 266536 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 266536 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 266536 T1075 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 266537 T1075 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371916096415/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/
[junit4:junit4] 2> 266537 T1075 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 266537 T1075 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 266537 T1075 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2
[junit4:junit4] 2> 266538 T1075 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index/
[junit4:junit4] 2> 266538 T1075 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 266538 T1075 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index
[junit4:junit4] 2> 266541 T1075 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 266541 T1075 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 266543 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 266543 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 266543 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 266544 T1075 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 266544 T1075 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 266544 T1075 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 266544 T1075 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 266545 T1075 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 266545 T1075 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 266546 T1075 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 266549 T1075 oass.SolrIndexSearcher.<init> Opening Searcher@5df84050 main
[junit4:junit4] 2> 266549 T1075 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 266550 T1075 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 266553 T1076 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5df84050 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 266554 T1075 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 266555 T1075 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46291/vzz/ut collection:collection1 shard:shard1
[junit4:junit4] 2> 266555 T1075 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 266561 T1075 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 266564 T1075 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 266564 T1075 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 266564 T1075 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46291/vzz/ut/collection1/
[junit4:junit4] 2> 266564 T1075 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 266565 T1075 oasc.SyncStrategy.syncToMe http://127.0.0.1:46291/vzz/ut/collection1/ has no replicas
[junit4:junit4] 2> 266565 T1075 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46291/vzz/ut/collection1/
[junit4:junit4] 2> 266565 T1075 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 266950 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 266957 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 266957 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 266958 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 266957 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 266973 T1075 oasc.ZkController.register We are http://127.0.0.1:46291/vzz/ut/collection1/ and leader is http://127.0.0.1:46291/vzz/ut/collection1/
[junit4:junit4] 2> 266973 T1075 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46291/vzz/ut
[junit4:junit4] 2> 266973 T1075 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 266973 T1075 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 266973 T1075 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 266975 T1075 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 266977 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 266977 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 266977 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 267060 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 267062 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37348
[junit4:junit4] 2> 267062 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 267063 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 267063 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445
[junit4:junit4] 2> 267063 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/solr.xml
[junit4:junit4] 2> 267063 T1016 oasc.CoreContainer.<init> New CoreContainer 782065824
[junit4:junit4] 2> 267064 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/'
[junit4:junit4] 2> 267064 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/'
[junit4:junit4] 2> 267116 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 267116 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 267117 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 267117 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 267117 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 267117 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 267117 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 267118 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 267118 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 267118 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 267122 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 267122 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 267122 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 267123 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 267125 T1088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fa971fe name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 267125 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 267127 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 267130 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 267132 T1090 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@477d64ed name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 267132 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 267135 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 268138 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37348_vzz%2Fut
[junit4:junit4] 2> 268139 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37348_vzz%2Fut
[junit4:junit4] 2> 268141 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 268143 T1074 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 268143 T1090 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 268143 T1045 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 268143 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268143 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268143 T1058 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 268144 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268145 T1038 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 268149 T1091 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 268149 T1091 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 268465 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 268466 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46291/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46291_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:46291_vzz%2Fut"}
[junit4:junit4] 2> 268470 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37348/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37348_vzz%2Fut"}
[junit4:junit4] 2> 268470 T1039 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 268470 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 268475 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268475 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268475 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268475 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 268475 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 269150 T1091 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/collection1
[junit4:junit4] 2> 269150 T1091 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 269151 T1091 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 269151 T1091 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 269152 T1091 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/collection1/'
[junit4:junit4] 2> 269153 T1091 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/collection1/lib/README' to classloader
[junit4:junit4] 2> 269153 T1091 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 269177 T1091 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 269213 T1091 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 269214 T1091 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 269217 T1091 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 269551 T1091 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 269556 T1091 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 269558 T1091 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 269577 T1091 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 269580 T1091 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 269583 T1091 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 269584 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 269584 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 269584 T1091 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 269585 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 269585 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 269585 T1091 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 269586 T1091 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371916099445/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty3/
[junit4:junit4] 2> 269586 T1091 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 269586 T1091 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 269586 T1091 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty3
[junit4:junit4] 2> 269587 T1091 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty3/index/
[junit4:junit4] 2> 269587 T1091 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 269587 T1091 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty3/index
[junit4:junit4] 2> 269590 T1091 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 269591 T1091 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 269592 T1091 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 269592 T1091 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 269593 T1091 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 269593 T1091 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 269594 T1091 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 269594 T1091 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 269594 T1091 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 269594 T1091 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 269594 T1091 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 269596 T1091 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 269599 T1091 oass.SolrIndexSearcher.<init> Opening Searcher@7e9e9b3b main
[junit4:junit4] 2> 269599 T1091 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 269599 T1091 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 269604 T1092 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e9e9b3b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 269606 T1091 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 269606 T1091 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37348/vzz/ut collection:collection1 shard:shard2
[junit4:junit4] 2> 269607 T1091 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 269612 T1091 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 269614 T1091 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 269614 T1091 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 269614 T1091 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37348/vzz/ut/collection1/
[junit4:junit4] 2> 269614 T1091 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 269615 T1091 oasc.SyncStrategy.syncToMe http://127.0.0.1:37348/vzz/ut/collection1/ has no replicas
[junit4:junit4] 2> 269615 T1091 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37348/vzz/ut/collection1/
[junit4:junit4] 2> 269615 T1091 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 269982 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 269990 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 269990 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 269990 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 269990 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 269990 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 270020 T1091 oasc.ZkController.register We are http://127.0.0.1:37348/vzz/ut/collection1/ and leader is http://127.0.0.1:37348/vzz/ut/collection1/
[junit4:junit4] 2> 270021 T1091 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37348/vzz/ut
[junit4:junit4] 2> 270021 T1091 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 270021 T1091 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 270021 T1091 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 270023 T1091 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 270024 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 270024 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 270024 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 270098 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 270100 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48205
[junit4:junit4] 2> 270101 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 270101 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 270101 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490
[junit4:junit4] 2> 270102 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/solr.xml
[junit4:junit4] 2> 270102 T1016 oasc.CoreContainer.<init> New CoreContainer 1409425530
[junit4:junit4] 2> 270102 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/'
[junit4:junit4] 2> 270102 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/'
[junit4:junit4] 2> 270144 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 270145 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 270145 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 270146 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 270147 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 270147 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 270147 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 270147 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 270148 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 270148 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 270151 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 270152 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 270152 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 270152 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 270154 T1104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@724e1205 name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 270154 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 270156 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 270167 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 270169 T1106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12ebbab6 name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 270169 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 270172 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 271176 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48205_vzz%2Fut
[junit4:junit4] 2> 271177 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48205_vzz%2Fut
[junit4:junit4] 2> 271179 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 271179 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 271179 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 271181 T1074 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 271181 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271181 T1045 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 271181 T1106 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 271181 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271183 T1038 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 271183 T1090 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 271183 T1058 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 271188 T1107 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 271188 T1107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 271496 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 271497 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37348/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:37348_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:37348_vzz%2Fut"}
[junit4:junit4] 2> 271501 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48205/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48205_vzz%2Fut"}
[junit4:junit4] 2> 271501 T1039 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 271501 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 271505 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271505 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271505 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271505 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271505 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 271505 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 272189 T1107 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/collection1
[junit4:junit4] 2> 272190 T1107 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 272190 T1107 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 272191 T1107 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 272192 T1107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/collection1/'
[junit4:junit4] 2> 272192 T1107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/collection1/lib/README' to classloader
[junit4:junit4] 2> 272193 T1107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 272215 T1107 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 272250 T1107 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 272252 T1107 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 272255 T1107 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 272568 T1107 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 272573 T1107 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 272575 T1107 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 272589 T1107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 272592 T1107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 272594 T1107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 272596 T1107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 272596 T1107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 272596 T1107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 272597 T1107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 272597 T1107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 272598 T1107 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 272598 T1107 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371916102490/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/
[junit4:junit4] 2> 272598 T1107 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 272598 T1107 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 272599 T1107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4
[junit4:junit4] 2> 272599 T1107 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index/
[junit4:junit4] 2> 272599 T1107 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 272599 T1107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index
[junit4:junit4] 2> 272603 T1107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 272603 T1107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 272605 T1107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 272605 T1107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 272606 T1107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 272606 T1107 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 272607 T1107 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 272607 T1107 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 272607 T1107 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 272608 T1107 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 272608 T1107 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 272610 T1107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 272615 T1107 oass.SolrIndexSearcher.<init> Opening Searcher@5415fa3b main
[junit4:junit4] 2> 272616 T1107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 272616 T1107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 272619 T1108 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5415fa3b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 272621 T1107 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 272621 T1107 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48205/vzz/ut collection:collection1 shard:shard3
[junit4:junit4] 2> 272625 T1107 oasc.ZkController.register We are http://127.0.0.1:48205/vzz/ut/collection1/ and leader is http://127.0.0.1:55690/vzz/ut/collection1/
[junit4:junit4] 2> 272625 T1107 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48205/vzz/ut
[junit4:junit4] 2> 272625 T1107 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 272625 T1107 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2421 name=collection1 org.apache.solr.core.SolrCore@544c844b url=http://127.0.0.1:48205/vzz/ut/collection1 node=127.0.0.1:48205_vzz%2Fut C2421_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, base_url=http://127.0.0.1:48205/vzz/ut, node_name=127.0.0.1:48205_vzz%2Fut}
[junit4:junit4] 2> 272625 T1109 C2421 P48205 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 272626 T1109 C2421 P48205 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 272626 T1107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 272626 T1109 C2421 P48205 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 272626 T1109 C2421 P48205 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 272627 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 272627 T1109 C2421 P48205 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 272627 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 272628 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 272632 T1051 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:48205_vzz%2Fut_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 272695 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 272697 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48136
[junit4:junit4] 2> 272698 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 272698 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 272698 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092
[junit4:junit4] 2> 272699 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/solr.xml
[junit4:junit4] 2> 272699 T1016 oasc.CoreContainer.<init> New CoreContainer 1252264
[junit4:junit4] 2> 272699 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/'
[junit4:junit4] 2> 272699 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/'
[junit4:junit4] 2> 272745 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 272745 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 272745 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 272746 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 272746 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 272746 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 272746 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 272746 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 272747 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 272747 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 272750 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 272751 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 272751 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 272752 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 272753 T1121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@342aada name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 272754 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 272755 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 272759 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 272760 T1123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39d0ed4c name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 272760 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 272763 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 273012 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 273013 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48205/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48205_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48205_vzz%2Fut"}
[junit4:junit4] 2> 273017 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273018 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273018 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273018 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273018 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273018 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273019 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273632 T1051 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:48205_vzz%2Fut_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 273633 T1051 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={checkLive=true&state=recovering&core=collection1&action=PREPRECOVERY&wt=javabin&onlyIfLeader=true&coreNodeName=127.0.0.1:48205_vzz%252Fut_collection1&version=2&nodeName=127.0.0.1:48205_vzz%252Fut} status=0 QTime=1001
[junit4:junit4] 2> 273766 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48136_vzz%2Fut
[junit4:junit4] 2> 273767 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48136_vzz%2Fut
[junit4:junit4] 2> 273769 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273769 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273770 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273769 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273770 T1074 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273770 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 273770 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 273771 T1058 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273771 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 273772 T1045 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273773 T1106 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273774 T1090 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273774 T1038 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273776 T1123 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 273781 T1124 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 273781 T1124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 274523 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 274524 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48136/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48136_vzz%2Fut"}
[junit4:junit4] 2> 274524 T1039 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 274524 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 274529 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274529 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274529 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274529 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274529 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274530 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274530 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 274782 T1124 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/collection1
[junit4:junit4] 2> 274782 T1124 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 274783 T1124 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 274783 T1124 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 274784 T1124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/collection1/'
[junit4:junit4] 2> 274785 T1124 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/collection1/lib/README' to classloader
[junit4:junit4] 2> 274785 T1124 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 274812 T1124 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 274850 T1124 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 274851 T1124 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 274855 T1124 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 275235 T1124 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 275243 T1124 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 275247 T1124 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 275274 T1124 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 275277 T1124 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 275290 T1124 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 275291 T1124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 275292 T1124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 275292 T1124 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 275294 T1124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 275294 T1124 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 275294 T1124 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 275295 T1124 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371916105092/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/
[junit4:junit4] 2> 275295 T1124 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 275295 T1124 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 275296 T1124 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5
[junit4:junit4] 2> 275296 T1124 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index/
[junit4:junit4] 2> 275296 T1124 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 275297 T1124 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index
[junit4:junit4] 2> 275301 T1124 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 275301 T1124 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 275308 T1124 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 275308 T1124 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 275310 T1124 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 275311 T1124 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 275311 T1124 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 275311 T1124 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 275312 T1124 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 275312 T1124 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 275313 T1124 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 275314 T1124 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 275318 T1124 oass.SolrIndexSearcher.<init> Opening Searcher@29c6cd70 main
[junit4:junit4] 2> 275319 T1124 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 275319 T1124 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 275323 T1125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29c6cd70 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 275325 T1124 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 275325 T1124 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48136/vzz/ut collection:collection1 shard:shard1
[junit4:junit4] 2> 275329 T1124 oasc.ZkController.register We are http://127.0.0.1:48136/vzz/ut/collection1/ and leader is http://127.0.0.1:46291/vzz/ut/collection1/
[junit4:junit4] 2> 275329 T1124 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48136/vzz/ut
[junit4:junit4] 2> 275329 T1124 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 275330 T1124 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2422 name=collection1 org.apache.solr.core.SolrCore@6052b748 url=http://127.0.0.1:48136/vzz/ut/collection1 node=127.0.0.1:48136_vzz%2Fut C2422_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, base_url=http://127.0.0.1:48136/vzz/ut, node_name=127.0.0.1:48136_vzz%2Fut}
[junit4:junit4] 2> 275330 T1126 C2422 P48136 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 275331 T1126 C2422 P48136 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 275331 T1124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 275331 T1126 C2422 P48136 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 275331 T1126 C2422 P48136 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 275332 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 275332 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 275332 T1126 C2422 P48136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 275333 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 275341 T1070 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:48136_vzz%2Fut_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 275446 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 275449 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40875
[junit4:junit4] 2> 275449 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 275450 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 275450 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801
[junit4:junit4] 2> 275450 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/solr.xml
[junit4:junit4] 2> 275451 T1016 oasc.CoreContainer.<init> New CoreContainer 1515793290
[junit4:junit4] 2> 275451 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/'
[junit4:junit4] 2> 275452 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/'
[junit4:junit4] 2> 275516 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 275517 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 275517 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 275517 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 275518 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 275518 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 275518 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 275519 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 275519 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 275520 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 275525 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 275525 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 275526 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 275527 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 275529 T1138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57e7a36 name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 275530 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 275543 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 275548 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 275568 T1140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7841e368 name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 275569 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 275573 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2423 name=collection1 org.apache.solr.core.SolrCore@544c844b url=http://127.0.0.1:48205/vzz/ut/collection1 node=127.0.0.1:48205_vzz%2Fut C2423_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, base_url=http://127.0.0.1:48205/vzz/ut, node_name=127.0.0.1:48205_vzz%2Fut}
[junit4:junit4] 2> 275634 T1109 C2423 P48205 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55690/vzz/ut/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 275634 T1109 C2423 P48205 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48205/vzz/ut START replicas=[http://127.0.0.1:55690/vzz/ut/collection1/] nUpdates=100
[junit4:junit4] 2> 275635 T1109 C2423 P48205 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 275638 T1109 C2423 P48205 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 275638 T1109 C2423 P48205 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 275638 T1109 C2423 P48205 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 275638 T1109 C2423 P48205 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 275639 T1109 C2423 P48205 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55690/vzz/ut/collection1/. core=collection1
[junit4:junit4] 2> 275639 T1109 C2423 P48205 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2424 name=collection1 org.apache.solr.core.SolrCore@3905d10b url=http://127.0.0.1:55690/vzz/ut/collection1 node=127.0.0.1:55690_vzz%2Fut C2424_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, base_url=http://127.0.0.1:55690/vzz/ut, node_name=127.0.0.1:55690_vzz%2Fut, leader=true}
[junit4:junit4] 2> 275656 T1051 C2424 P55690 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/get params={qt=/get&wt=javabin&distrib=false&version=2&getVersions=100} status=0 QTime=0
[junit4:junit4] 2> 275656 T1053 C2424 P55690 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 275659 T1053 C2424 P55690 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 275660 T1053 C2424 P55690 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 275668 T1053 C2424 P55690 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 275668 T1053 C2424 P55690 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 275670 T1053 C2424 P55690 oass.SolrIndexSearcher.<init> Opening Searcher@691465df realtime
[junit4:junit4] 2> 275671 T1053 C2424 P55690 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 275671 T1053 C2424 P55690 oasup.LogUpdateProcessor.finish [collection1] webapp=/vzz/ut path=/update params={waitSearcher=true&softCommit=false&wt=javabin&commit_end_point=true&openSearcher=false&version=2&commit=true} {commit=} 0 15
[junit4:junit4] 2> ASYNC NEW_CORE C2425 name=collection1 org.apache.solr.core.SolrCore@544c844b url=http://127.0.0.1:48205/vzz/ut/collection1 node=127.0.0.1:48205_vzz%2Fut C2425_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, base_url=http://127.0.0.1:48205/vzz/ut, node_name=127.0.0.1:48205_vzz%2Fut}
[junit4:junit4] 2> 275672 T1109 C2425 P48205 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 275672 T1109 C2425 P48205 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 275676 T1051 C2424 P55690 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 275676 T1051 C2424 P55690 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=1
[junit4:junit4] 2> 275677 T1109 C2425 P48205 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 275677 T1109 C2425 P48205 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 275677 T1109 C2425 P48205 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 275680 T1053 C2424 P55690 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/replication params={qt=/replication&wt=javabin&version=2&generation=2&command=filelist} status=0 QTime=1
[junit4:junit4] 2> 275680 T1109 C2425 P48205 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 275681 T1109 C2425 P48205 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index.20130622104828143
[junit4:junit4] 2> 275681 T1109 C2425 P48205 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index.20130622104828143 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d19963f fullCopy=false
[junit4:junit4] 2> 275684 T1052 C2424 P55690 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/replication params={qt=/replication&wt=filestream&file=segments_2&generation=2&command=filecontent&checksum=true} status=0 QTime=0
[junit4:junit4] 2> 275685 T1109 C2425 P48205 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 275699 T1109 C2425 P48205 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 275699 T1109 C2425 P48205 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 275702 T1109 C2425 P48205 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 275702 T1109 C2425 P48205 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 275703 T1109 C2425 P48205 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 275703 T1109 C2425 P48205 oass.SolrIndexSearcher.<init> Opening Searcher@c4cd652 main
[junit4:junit4] 2> 275704 T1108 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c4cd652 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 275705 T1109 C2425 P48205 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index.20130622104828143 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index.20130622104828143;done=true>>]
[junit4:junit4] 2> 275705 T1109 C2425 P48205 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index.20130622104828143
[junit4:junit4] 2> 275705 T1109 C2425 P48205 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty4/index.20130622104828143
[junit4:junit4] 2> 275706 T1109 C2425 P48205 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 275706 T1109 C2425 P48205 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 275706 T1109 C2425 P48205 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 275707 T1109 C2425 P48205 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 275709 T1109 C2425 P48205 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 276034 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 276035 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48136/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48136_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48136_vzz%2Fut"}
[junit4:junit4] 2> 276039 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48205/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48205_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48205_vzz%2Fut"}
[junit4:junit4] 2> 276058 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276058 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276059 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276059 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276059 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276059 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276059 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276059 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276343 T1070 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:48136_vzz%2Fut_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 276344 T1070 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={checkLive=true&state=recovering&core=collection1&action=PREPRECOVERY&wt=javabin&onlyIfLeader=true&coreNodeName=127.0.0.1:48136_vzz%252Fut_collection1&version=2&nodeName=127.0.0.1:48136_vzz%252Fut} status=0 QTime=1003
[junit4:junit4] 2> 276580 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40875_vzz%2Fut
[junit4:junit4] 2> 276581 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40875_vzz%2Fut
[junit4:junit4] 2> 276583 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276583 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276584 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276584 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276583 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 276585 T1045 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276586 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 276586 T1038 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276586 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 276586 T1058 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276587 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 276588 T1090 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276589 T1123 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276589 T1140 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276589 T1106 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276589 T1074 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 276594 T1143 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 276595 T1143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 277569 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 277569 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40875/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40875_vzz%2Fut"}
[junit4:junit4] 2> 277570 T1039 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 277570 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 277574 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277574 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 277596 T1143 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/collection1
[junit4:junit4] 2> 277596 T1143 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 277597 T1143 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 277597 T1143 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 277598 T1143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/collection1/'
[junit4:junit4] 2> 277599 T1143 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/collection1/lib/README' to classloader
[junit4:junit4] 2> 277599 T1143 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 277625 T1143 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 277661 T1143 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 277662 T1143 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 277666 T1143 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 278000 T1143 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 278005 T1143 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 278007 T1143 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 278023 T1143 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 278025 T1143 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 278028 T1143 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 278029 T1143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 278029 T1143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 278029 T1143 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 278030 T1143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 278030 T1143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 278031 T1143 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 278031 T1143 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371916107801/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty6/
[junit4:junit4] 2> 278031 T1143 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5fb9589
[junit4:junit4] 2> 278031 T1143 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 278032 T1143 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty6
[junit4:junit4] 2> 278032 T1143 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty6/index/
[junit4:junit4] 2> 278032 T1143 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 278032 T1143 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty6/index
[junit4:junit4] 2> 278035 T1143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 278035 T1143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 278037 T1143 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 278037 T1143 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 278037 T1143 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 278038 T1143 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 278038 T1143 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 278038 T1143 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 278038 T1143 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 278039 T1143 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 278039 T1143 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 278040 T1143 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 278043 T1143 oass.SolrIndexSearcher.<init> Opening Searcher@54ad2fd main
[junit4:junit4] 2> 278044 T1143 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 278044 T1143 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 278047 T1144 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@54ad2fd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 278049 T1143 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 278049 T1143 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40875/vzz/ut collection:collection1 shard:shard2
[junit4:junit4] 2> 278052 T1143 oasc.ZkController.register We are http://127.0.0.1:40875/vzz/ut/collection1/ and leader is http://127.0.0.1:37348/vzz/ut/collection1/
[junit4:junit4] 2> 278052 T1143 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40875/vzz/ut
[junit4:junit4] 2> 278052 T1143 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 278052 T1143 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2426 name=collection1 org.apache.solr.core.SolrCore@6a0f5f06 url=http://127.0.0.1:40875/vzz/ut/collection1 node=127.0.0.1:40875_vzz%2Fut C2426_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, base_url=http://127.0.0.1:40875/vzz/ut, node_name=127.0.0.1:40875_vzz%2Fut}
[junit4:junit4] 2> 278052 T1145 C2426 P40875 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 278053 T1145 C2426 P40875 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 278053 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 278053 T1145 C2426 P40875 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 278053 T1145 C2426 P40875 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 278054 T1016 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 278054 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 278054 T1145 C2426 P40875 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 278055 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 278061 T1085 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:40875_vzz%2Fut_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 278133 T1016 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 278136 T1016 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56099
[junit4:junit4] 2> 278136 T1016 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 278136 T1016 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 278137 T1016 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522
[junit4:junit4] 2> 278137 T1016 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/solr.xml
[junit4:junit4] 2> 278137 T1016 oasc.CoreContainer.<init> New CoreContainer 2062262075
[junit4:junit4] 2> 278138 T1016 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/'
[junit4:junit4] 2> 278138 T1016 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/'
[junit4:junit4] 2> 278194 T1016 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 278194 T1016 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 278195 T1016 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 278195 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 278195 T1016 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 278195 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 278196 T1016 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 278196 T1016 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 278196 T1016 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 278196 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 278200 T1016 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 278201 T1016 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35160/solr
[junit4:junit4] 2> 278201 T1016 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 278202 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 278204 T1157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c937312 name:ZooKeeperConnection Watcher:127.0.0.1:35160 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 278204 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 278206 T1016 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 278209 T1016 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 278211 T1159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bde62bf name:ZooKeeperConnection Watcher:127.0.0.1:35160/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 278211 T1016 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 278215 T1016 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2427 name=collection1 org.apache.solr.core.SolrCore@6052b748 url=http://127.0.0.1:48136/vzz/ut/collection1 node=127.0.0.1:48136_vzz%2Fut C2427_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, base_url=http://127.0.0.1:48136/vzz/ut, node_name=127.0.0.1:48136_vzz%2Fut}
[junit4:junit4] 2> 278345 T1126 C2427 P48136 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46291/vzz/ut/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 278345 T1126 C2427 P48136 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48136/vzz/ut START replicas=[http://127.0.0.1:46291/vzz/ut/collection1/] nUpdates=100
[junit4:junit4] 2> 278345 T1126 C2427 P48136 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 278345 T1126 C2427 P48136 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 278346 T1126 C2427 P48136 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 278346 T1126 C2427 P48136 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 278346 T1126 C2427 P48136 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 278346 T1126 C2427 P48136 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46291/vzz/ut/collection1/. core=collection1
[junit4:junit4] 2> 278346 T1126 C2427 P48136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2428 name=collection1 org.apache.solr.core.SolrCore@68e74df url=http://127.0.0.1:46291/vzz/ut/collection1 node=127.0.0.1:46291_vzz%2Fut C2428_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, base_url=http://127.0.0.1:46291/vzz/ut, node_name=127.0.0.1:46291_vzz%2Fut, leader=true}
[junit4:junit4] 2> 278352 T1070 C2428 P46291 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/get params={qt=/get&wt=javabin&distrib=false&version=2&getVersions=100} status=0 QTime=0
[junit4:junit4] 2> 278355 T1067 C2428 P46291 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 278359 T1067 C2428 P46291 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 278359 T1067 C2428 P46291 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 278368 T1067 C2428 P46291 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 278368 T1067 C2428 P46291 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 278369 T1067 C2428 P46291 oass.SolrIndexSearcher.<init> Opening Searcher@2dd00a96 realtime
[junit4:junit4] 2> 278369 T1067 C2428 P46291 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 278369 T1067 C2428 P46291 oasup.LogUpdateProcessor.finish [collection1] webapp=/vzz/ut path=/update params={waitSearcher=true&softCommit=false&wt=javabin&commit_end_point=true&openSearcher=false&version=2&commit=true} {commit=} 0 14
[junit4:junit4] 2> 278370 T1126 C2427 P48136 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 278371 T1126 C2427 P48136 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 278372 T1069 C2428 P46291 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 278373 T1069 C2428 P46291 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=1
[junit4:junit4] 2> 278373 T1126 C2427 P48136 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 278373 T1126 C2427 P48136 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 278374 T1126 C2427 P48136 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 278376 T1070 C2428 P46291 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/replication params={qt=/replication&wt=javabin&version=2&generation=2&command=filelist} status=0 QTime=1
[junit4:junit4] 2> 278376 T1126 C2427 P48136 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 278377 T1126 C2427 P48136 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index.20130622104830839
[junit4:junit4] 2> 278377 T1126 C2427 P48136 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index.20130622104830839 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60381bc fullCopy=false
[junit4:junit4] 2> 278380 T1067 C2428 P46291 oasc.SolrCore.execute [collection1] webapp=/vzz/ut path=/replication params={qt=/replication&wt=filestream&file=segments_2&generation=2&command=filecontent&checksum=true} status=0 QTime=0
[junit4:junit4] 2> 278381 T1126 C2427 P48136 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 278389 T1126 C2427 P48136 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 278389 T1126 C2427 P48136 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 278391 T1126 C2427 P48136 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 278391 T1126 C2427 P48136 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 278392 T1126 C2427 P48136 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 278392 T1126 C2427 P48136 oass.SolrIndexSearcher.<init> Opening Searcher@3eb11d6 main
[junit4:junit4] 2> 278393 T1125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3eb11d6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 278393 T1126 C2427 P48136 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index.20130622104830839 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index.20130622104830839;done=true>>]
[junit4:junit4] 2> 278393 T1126 C2427 P48136 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index.20130622104830839
[junit4:junit4] 2> 278393 T1126 C2427 P48136 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371916089793/jetty5/index.20130622104830839
[junit4:junit4] 2> 278394 T1126 C2427 P48136 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 278394 T1126 C2427 P48136 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 278394 T1126 C2427 P48136 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 278394 T1126 C2427 P48136 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 278396 T1126 C2427 P48136 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 279082 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 279083 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40875/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40875_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:40875_vzz%2Fut"}
[junit4:junit4] 2> 279087 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48136/vzz/ut",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48136_vzz%2Fut_collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48136_vzz%2Fut"}
[junit4:junit4] 2> 279112 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279112 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279219 T1016 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56099_vzz%2Fut
[junit4:junit4] 2> 279220 T1016 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56099_vzz%2Fut
[junit4:junit4] 2> 279222 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279222 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279222 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279222 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279222 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279222 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279222 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 279223 T1159 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279223 T1159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 279223 T1058 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279224 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 279227 T1074 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279227 T1090 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279227 T1045 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279228 T1106 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279228 T1123 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279228 T1038 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279228 T1140 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 279234 T1162 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 279234 T1162 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 280062 T1085 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:40875_vzz%2Fut_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 280062 T1085 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={checkLive=true&state=recovering&core=collection1&action=PREPRECOVERY&wt=javabin&onlyIfLeader=true&coreNodeName=127.0.0.1:40875_vzz%252Fut_collection1&version=2&nodeName=127.0.0.1:40875_vzz%252Fut} status=0 QTime=2001
[junit4:junit4] 2> 280620 T1039 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 280621 T1039 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56099/vzz/ut",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56099_vzz%2Fut"}
[junit4:junit4] 2> 280621 T1039 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 280621 T1039 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 280626 T1123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280627 T1038 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1159 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1045 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 280626 T1058 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 281236 T1162 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/collection1
[junit4:junit4] 2> 281236 T1162 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 281237 T1162 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 281237 T1162 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 281239 T1162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/collection1/'
[junit4:junit4] 2> 281239 T1162 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/collection1/lib/README' to classloader
[junit4:junit4] 2> 281239 T1162 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371916110522/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 281266 T1162 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 281294 T1162 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 281295 T1162 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 281298 T1162 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 281674 T1162 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 281679 T1162 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 281681 T1162 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 281697 T1162 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 281700 T1162 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 281703 T1162 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 281704 T1162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 281705 T1162 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 281705 T1162 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[...truncated too long message...]
1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89909907894566915-127.0.0.1:57545__cjv%2Fx-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89909907894566926-127.0.0.1:44408__cjv%2Fx-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89909907894566924-127.0.0.1:49856__cjv%2Fx-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89909907894566922-127.0.0.1:60443__cjv%2Fx-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89909907894566915-127.0.0.1:57545__cjv%2Fx-n_0000000000 (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:57545__cjv%2Fx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:44408__cjv%2Fx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:60443__cjv%2Fx (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:49856__cjv%2Fx (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:57545__cjv%2Fx",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:57545/_cjv/x"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89909907894566915-127.0.0.1:57545__cjv%2Fx_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:60443__cjv%2Fx",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60443/_cjv/x"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89909907894566922-127.0.0.1:60443__cjv%2Fx_collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89909907894566926-127.0.0.1:44408__cjv%2Fx_collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89909907894566924-127.0.0.1:49856__cjv%2Fx_collection1-n_0000000003 (0)
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=E4C9D5F38A804DC7 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=no -Dtests.timezone=America/Atka -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 101s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:44408/_cjv/x/collection1lastClient and got 241 from http://127.0.0.1:60443/_cjv/x/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([E4C9D5F38A804DC7:652F5BEBFDDF2DFB]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 588939 T2646 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 101152 T2645 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4] 2> 589062 T2721 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 589062 T2721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 589062 T2721 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 590145 T2738 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 590145 T2738 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 590145 T2738 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {text=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), intDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=PostingsFormat(name=TestBloomFilteredLucene41Postings), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=Pulsing41(freqCutoff=3 minBlockSize=82 maxBlockSize=209), rnd_b=MockVariableIntBlock(baseBlockSize=127), id=MockVariableIntBlock(baseBlockSize=127), range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_si=Pulsing41(freqCutoff=3 minBlockSize=82 maxBlockSize=209), a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=Pulsing41(freqCutoff=3 minBlockSize=82 maxBlockSize=209), range_facet_sl=PostingsFormat(name=Direct)}, docValues:{timestamp=DocValuesFormat(name=Lucene42)}, sim=DefaultSimilarity, locale=no, timezone=America/Atka
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=75417312,total=262217728
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestCloudManagedSchemaAddField, SOLR749Test, TestLFUCache, ZkCLITest, SuggesterFSTTest, TestShardHandlerFactory, LukeRequestHandlerTest, FastVectorHighlighterTest, TestFastWriter, SpellingQueryConverterTest, SynonymTokenizerTest, OverseerCollectionProcessorTest, OverseerTest, TestRemoteStreaming, PolyFieldTest, TestQueryUtils, TestExtendedDismaxParser, TestPluginEnable, CircularListTest, DistributedTermsComponentTest, TestMaxScoreQueryParser, CacheHeaderTest, ResponseLogComponentTest, TestReversedWildcardFilterFactory, SliceStateTest, AliasIntegrationTest, PreAnalyzedFieldTest, TestSolrQueryParserDefaultOperatorResource, LeaderElectionTest, TestConfig, DirectSolrConnectionTest, BadComponentTest, RAMDirectoryFactoryTest, DocumentBuilderTest, DirectUpdateHandlerOptimizeTest, LegacyHTMLStripCharFilterTest, TestSurroundQueryParser, SimpleFacetsTest, AnalysisAfterCoreReloadTest, TestLRUCache, TestDocSet, SchemaVersionSpecificBehaviorTest, TermVectorComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestSolrIndexConfig, ResourceLoaderTest, PrimUtilsTest, MoreLikeThisHandlerTest, ChaosMonkeyNothingIsSafeTest, SystemInfoHandlerTest, UnloadDistributedZkTest, TestQuerySenderListener, WordBreakSolrSpellCheckerTest, IndexSchemaTest, TestRealTimeGet, MBeansHandlerTest, SpatialFilterTest, BasicFunctionalityTest, TestSolrDeletionPolicy2, TestFieldCollectionResource, BasicDistributedZkTest, SolrIndexSplitterTest, ReturnFieldsTest, TestPerFieldSimilarity, AlternateDirectoryTest, UpdateRequestProcessorFactoryTest, TestReplicationHandler, DateFieldTest, QueryParsingTest, ExternalFileFieldSortTest, TestRandomDVFaceting, SyncSliceTest]
[junit4:junit4] Completed on J0 in 102.42s, 1 test, 1 failure <<< FAILURES!
[...truncated 466 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 2 failures, 16 ignored (7 assumptions)
Total time: 36 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b93 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure