You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/23 17:58:43 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b93) - Build #
6202 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6202/
Java: 64bit/jdk1.8.0-ea-b93 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 26 from http://127.0.0.1:57492/h_dkw/r/collection1lastClient and got 25 from http://127.0.0.1:53431/h_dkw/r/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 26 from http://127.0.0.1:57492/h_dkw/r/collection1lastClient and got 25 from http://127.0.0.1:53431/h_dkw/r/collection1
at __randomizedtesting.SeedInfo.seed([502D08402E749A31:D1CB8658592BFA0D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9629 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 360306 T1053 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /h_dkw/r
[junit4:junit4] 2> 360312 T1053 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1372002403441
[junit4:junit4] 2> 360312 T1053 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 360313 T1054 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 360413 T1053 oasc.ZkTestServer.run start zk server on port:52305
[junit4:junit4] 2> 360414 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360433 T1060 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dc14416 name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360434 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360434 T1053 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 360439 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360440 T1062 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@306abde name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360440 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360440 T1053 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 360443 T1053 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 360446 T1053 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 360448 T1053 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 360454 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 360454 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 360458 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 360459 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 360462 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 360462 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 360465 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 360465 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 360469 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 360469 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 360472 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 360472 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 360475 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 360475 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 360478 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 360478 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 360481 T1053 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 360481 T1053 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 360550 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 360574 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55279
[junit4:junit4] 2> 360575 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 360575 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 360575 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613
[junit4:junit4] 2> 360576 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/solr.xml
[junit4:junit4] 2> 360576 T1053 oasc.CoreContainer.<init> New CoreContainer 906077753
[junit4:junit4] 2> 360576 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/'
[junit4:junit4] 2> 360576 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/'
[junit4:junit4] 2> 360627 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 360627 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 360627 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 360627 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 360628 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 360628 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 360628 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 360628 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 360628 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 360629 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 360632 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 360633 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 360633 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 360633 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360635 T1073 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@255eb50e name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360636 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360638 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 360641 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360643 T1075 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69fda0ec name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360643 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360645 T1053 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 360649 T1053 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 360651 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 360653 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55279_h_dkw%2Fr
[junit4:junit4] 2> 360653 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55279_h_dkw%2Fr
[junit4:junit4] 2> 360656 T1053 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 360661 T1053 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 360664 T1053 oasc.Overseer.start Overseer (id=89915549512040451-127.0.0.1:55279_h_dkw%2Fr-n_0000000000) starting
[junit4:junit4] 2> 360668 T1053 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 360671 T1077 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 360671 T1053 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 360674 T1053 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 360675 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 360677 T1076 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 360680 T1078 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 360680 T1078 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 362181 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 362182 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55279/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:55279_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 362182 T1076 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 362185 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 362189 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 362682 T1078 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/collection1
[junit4:junit4] 2> 362682 T1078 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 362683 T1078 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 362683 T1078 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 362684 T1078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/collection1/'
[junit4:junit4] 2> 362684 T1078 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/collection1/lib/README' to classloader
[junit4:junit4] 2> 362685 T1078 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 362739 T1078 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 362813 T1078 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 362814 T1078 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 362834 T1078 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 363174 T1078 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 363174 T1078 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 363175 T1078 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 363181 T1078 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 363183 T1078 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 363201 T1078 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 363204 T1078 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 363207 T1078 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 363208 T1078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 363208 T1078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 363208 T1078 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 363209 T1078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 363209 T1078 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 363209 T1078 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 363216 T1078 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372002403613/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/control/data/
[junit4:junit4] 2> 363216 T1078 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 363216 T1078 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 363217 T1078 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/control/data
[junit4:junit4] 2> 363217 T1078 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/control/data/index/
[junit4:junit4] 2> 363217 T1078 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 363217 T1078 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/control/data/index
[junit4:junit4] 2> 363220 T1078 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 363220 T1078 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 363222 T1078 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 363222 T1078 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 363222 T1078 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 363223 T1078 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 363223 T1078 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 363223 T1078 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 363223 T1078 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 363224 T1078 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 363224 T1078 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 363225 T1078 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 363228 T1078 oass.SolrIndexSearcher.<init> Opening Searcher@7a9e2294 main
[junit4:junit4] 2> 363229 T1078 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 363229 T1078 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 363232 T1079 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a9e2294 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 363234 T1078 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 363234 T1078 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55279/h_dkw/r collection:control_collection shard:shard1
[junit4:junit4] 2> 363235 T1078 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 363241 T1078 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 363243 T1078 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 363244 T1078 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 363244 T1078 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55279/h_dkw/r/collection1/
[junit4:junit4] 2> 363244 T1078 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 363244 T1078 oasc.SyncStrategy.syncToMe http://127.0.0.1:55279/h_dkw/r/collection1/ has no replicas
[junit4:junit4] 2> 363244 T1078 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55279/h_dkw/r/collection1/
[junit4:junit4] 2> 363245 T1078 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 363696 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 363703 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 363751 T1078 oasc.ZkController.register We are http://127.0.0.1:55279/h_dkw/r/collection1/ and leader is http://127.0.0.1:55279/h_dkw/r/collection1/
[junit4:junit4] 2> 363751 T1078 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55279/h_dkw/r
[junit4:junit4] 2> 363752 T1078 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 363752 T1078 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 363752 T1078 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 363754 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 363755 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 363755 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 363756 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 363760 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 363762 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 363764 T1082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a1e678b name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 363764 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 363765 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 363767 T1053 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 363834 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 363835 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34240
[junit4:junit4] 2> 363836 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 363836 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 363836 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897
[junit4:junit4] 2> 363837 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/solr.xml
[junit4:junit4] 2> 363837 T1053 oasc.CoreContainer.<init> New CoreContainer 278457560
[junit4:junit4] 2> 363837 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/'
[junit4:junit4] 2> 363837 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/'
[junit4:junit4] 2> 363880 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 363880 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 363880 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 363881 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 363881 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 363881 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 363881 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 363881 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 363882 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 363882 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 363885 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 363886 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 363886 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 363887 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 363889 T1093 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20aa5ffc name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 363889 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 363892 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 363895 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 363897 T1095 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b2c63f4 name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 363898 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 363901 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 364904 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34240_h_dkw%2Fr
[junit4:junit4] 2> 364905 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34240_h_dkw%2Fr
[junit4:junit4] 2> 364908 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 364908 T1082 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 364909 T1095 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 364910 T1075 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 364914 T1096 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 364914 T1096 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 365207 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 365208 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55279/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:55279_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 365211 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34240/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:34240_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 365211 T1076 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 365211 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 365215 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365215 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365215 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365916 T1096 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/collection1
[junit4:junit4] 2> 365916 T1096 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 365917 T1096 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 365917 T1096 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 365918 T1096 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/collection1/'
[junit4:junit4] 2> 365919 T1096 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/collection1/lib/README' to classloader
[junit4:junit4] 2> 365919 T1096 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 365943 T1096 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 365976 T1096 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 365978 T1096 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 365985 T1096 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 366321 T1096 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 366322 T1096 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 366322 T1096 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 366331 T1096 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 366334 T1096 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 366348 T1096 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 366351 T1096 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 366354 T1096 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 366355 T1096 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 366355 T1096 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 366355 T1096 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 366356 T1096 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 366356 T1096 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 366356 T1096 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 366356 T1096 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372002406897/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/
[junit4:junit4] 2> 366356 T1096 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 366357 T1096 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 366357 T1096 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1
[junit4:junit4] 2> 366357 T1096 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index/
[junit4:junit4] 2> 366358 T1096 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 366358 T1096 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index
[junit4:junit4] 2> 366361 T1096 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 366361 T1096 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 366362 T1096 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 366363 T1096 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 366363 T1096 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 366363 T1096 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 366364 T1096 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 366364 T1096 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 366364 T1096 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 366365 T1096 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 366365 T1096 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 366366 T1096 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 366369 T1096 oass.SolrIndexSearcher.<init> Opening Searcher@1198674 main
[junit4:junit4] 2> 366370 T1096 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 366370 T1096 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 366374 T1097 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1198674 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 366376 T1096 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 366376 T1096 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34240/h_dkw/r collection:collection1 shard:shard1
[junit4:junit4] 2> 366377 T1096 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 366386 T1096 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 366389 T1096 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 366389 T1096 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 366389 T1096 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34240/h_dkw/r/collection1/
[junit4:junit4] 2> 366389 T1096 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 366390 T1096 oasc.SyncStrategy.syncToMe http://127.0.0.1:34240/h_dkw/r/collection1/ has no replicas
[junit4:junit4] 2> 366390 T1096 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34240/h_dkw/r/collection1/
[junit4:junit4] 2> 366390 T1096 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 366721 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 366728 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 366728 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 366728 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 366746 T1096 oasc.ZkController.register We are http://127.0.0.1:34240/h_dkw/r/collection1/ and leader is http://127.0.0.1:34240/h_dkw/r/collection1/
[junit4:junit4] 2> 366746 T1096 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34240/h_dkw/r
[junit4:junit4] 2> 366746 T1096 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 366746 T1096 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 366747 T1096 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 366748 T1096 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 366750 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 366750 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 366750 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 366820 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 366823 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54594
[junit4:junit4] 2> 366823 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 366824 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 366824 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883
[junit4:junit4] 2> 366824 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/solr.xml
[junit4:junit4] 2> 366824 T1053 oasc.CoreContainer.<init> New CoreContainer 917693301
[junit4:junit4] 2> 366825 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/'
[junit4:junit4] 2> 366825 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/'
[junit4:junit4] 2> 366867 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 366868 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 366868 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 366868 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 366868 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 366869 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 366869 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 366869 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 366869 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 366870 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 366873 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 366873 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 366874 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 366874 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366876 T1109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ce42ebd name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366877 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366878 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 366882 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366883 T1111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@611b0371 name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366883 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366887 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 367889 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54594_h_dkw%2Fr
[junit4:junit4] 2> 367891 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54594_h_dkw%2Fr
[junit4:junit4] 2> 367892 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 367892 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 367892 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 367894 T1111 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367895 T1075 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367895 T1082 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367895 T1095 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367899 T1112 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 367899 T1112 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 368234 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 368235 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34240/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:34240_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 368238 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54594/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:54594_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 368239 T1076 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 368239 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 368263 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368263 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368263 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368263 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368901 T1112 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/collection1
[junit4:junit4] 2> 368901 T1112 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 368902 T1112 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 368902 T1112 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 368903 T1112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/collection1/'
[junit4:junit4] 2> 368904 T1112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/collection1/lib/README' to classloader
[junit4:junit4] 2> 368904 T1112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 368931 T1112 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 368960 T1112 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 368962 T1112 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 368966 T1112 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 369289 T1112 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 369290 T1112 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 369290 T1112 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 369296 T1112 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 369298 T1112 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 369313 T1112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 369316 T1112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 369318 T1112 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 369319 T1112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 369319 T1112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 369320 T1112 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 369320 T1112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 369321 T1112 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 369321 T1112 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 369321 T1112 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372002409883/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/
[junit4:junit4] 2> 369321 T1112 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 369321 T1112 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 369322 T1112 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2
[junit4:junit4] 2> 369322 T1112 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index/
[junit4:junit4] 2> 369322 T1112 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 369322 T1112 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index
[junit4:junit4] 2> 369326 T1112 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 369326 T1112 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 369328 T1112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 369328 T1112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 369329 T1112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 369329 T1112 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 369329 T1112 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 369330 T1112 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 369330 T1112 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 369330 T1112 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 369330 T1112 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 369332 T1112 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 369335 T1112 oass.SolrIndexSearcher.<init> Opening Searcher@3da6291 main
[junit4:junit4] 2> 369335 T1112 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 369335 T1112 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 369340 T1113 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3da6291 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 369342 T1112 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 369342 T1112 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54594/h_dkw/r collection:collection1 shard:shard3
[junit4:junit4] 2> 369343 T1112 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 369348 T1112 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 369350 T1112 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 369350 T1112 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 369350 T1112 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54594/h_dkw/r/collection1/
[junit4:junit4] 2> 369350 T1112 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 369351 T1112 oasc.SyncStrategy.syncToMe http://127.0.0.1:54594/h_dkw/r/collection1/ has no replicas
[junit4:junit4] 2> 369351 T1112 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54594/h_dkw/r/collection1/
[junit4:junit4] 2> 369351 T1112 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 369770 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 369776 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369776 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369776 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369777 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369807 T1112 oasc.ZkController.register We are http://127.0.0.1:54594/h_dkw/r/collection1/ and leader is http://127.0.0.1:54594/h_dkw/r/collection1/
[junit4:junit4] 2> 369808 T1112 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54594/h_dkw/r
[junit4:junit4] 2> 369808 T1112 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 369808 T1112 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 369808 T1112 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 369814 T1112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 369815 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 369815 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 369816 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 369913 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 369915 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54034
[junit4:junit4] 2> 369916 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 369916 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 369917 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948
[junit4:junit4] 2> 369917 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/solr.xml
[junit4:junit4] 2> 369917 T1053 oasc.CoreContainer.<init> New CoreContainer 958943918
[junit4:junit4] 2> 369917 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/'
[junit4:junit4] 2> 369918 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/'
[junit4:junit4] 2> 369959 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 369960 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 369960 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 369960 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 369960 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 369960 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 369961 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 369961 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 369961 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 369961 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 369965 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 369966 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 369966 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 369967 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369968 T1125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@453bd837 name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369969 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369971 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 369974 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369977 T1127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@390c73f2 name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369977 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369980 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 370983 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54034_h_dkw%2Fr
[junit4:junit4] 2> 370985 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54034_h_dkw%2Fr
[junit4:junit4] 2> 370987 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370987 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370987 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370988 T1095 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370989 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 370989 T1127 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370990 T1082 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370990 T1111 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370990 T1075 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370995 T1128 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 370995 T1128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 371283 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 371284 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54594/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:54594_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 371286 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54034/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:54034_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 371287 T1076 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 371287 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 371303 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371303 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371303 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371303 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371303 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371997 T1128 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/collection1
[junit4:junit4] 2> 371997 T1128 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 371998 T1128 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 371998 T1128 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 371999 T1128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/collection1/'
[junit4:junit4] 2> 372000 T1128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/collection1/lib/README' to classloader
[junit4:junit4] 2> 372000 T1128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 372026 T1128 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 372065 T1128 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 372066 T1128 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 372071 T1128 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 372410 T1128 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 372411 T1128 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 372411 T1128 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 372417 T1128 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 372424 T1128 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 372437 T1128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 372440 T1128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 372443 T1128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 372444 T1128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 372444 T1128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 372444 T1128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 372445 T1128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 372445 T1128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 372446 T1128 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 372446 T1128 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372002412948/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty3/
[junit4:junit4] 2> 372446 T1128 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 372446 T1128 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 372447 T1128 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty3
[junit4:junit4] 2> 372447 T1128 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty3/index/
[junit4:junit4] 2> 372447 T1128 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 372447 T1128 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty3/index
[junit4:junit4] 2> 372456 T1128 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 372457 T1128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 372458 T1128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 372459 T1128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 372459 T1128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 372460 T1128 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 372460 T1128 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 372460 T1128 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 372460 T1128 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 372461 T1128 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 372461 T1128 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 372462 T1128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 372465 T1128 oass.SolrIndexSearcher.<init> Opening Searcher@4b4def74 main
[junit4:junit4] 2> 372466 T1128 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 372466 T1128 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 372470 T1129 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b4def74 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 372472 T1128 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 372472 T1128 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54034/h_dkw/r collection:collection1 shard:shard2
[junit4:junit4] 2> 372473 T1128 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 372506 T1128 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 372537 T1128 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 372537 T1128 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 372538 T1128 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54034/h_dkw/r/collection1/
[junit4:junit4] 2> 372538 T1128 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 372538 T1128 oasc.SyncStrategy.syncToMe http://127.0.0.1:54034/h_dkw/r/collection1/ has no replicas
[junit4:junit4] 2> 372538 T1128 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54034/h_dkw/r/collection1/
[junit4:junit4] 2> 372538 T1128 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 372816 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 372824 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372824 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372824 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372824 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372824 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372845 T1128 oasc.ZkController.register We are http://127.0.0.1:54034/h_dkw/r/collection1/ and leader is http://127.0.0.1:54034/h_dkw/r/collection1/
[junit4:junit4] 2> 372845 T1128 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54034/h_dkw/r
[junit4:junit4] 2> 372846 T1128 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 372846 T1128 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 372846 T1128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 372875 T1128 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 372876 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 372876 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 372877 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 372951 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 372954 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55152
[junit4:junit4] 2> 372955 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 372955 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 372955 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009
[junit4:junit4] 2> 372955 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/solr.xml
[junit4:junit4] 2> 372956 T1053 oasc.CoreContainer.<init> New CoreContainer 1148185587
[junit4:junit4] 2> 372956 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/'
[junit4:junit4] 2> 372956 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/'
[junit4:junit4] 2> 373003 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 373003 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 373003 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 373003 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 373004 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 373004 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 373004 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 373004 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 373004 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 373005 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 373008 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 373009 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 373009 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 373010 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 373024 T1141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15ec49f9 name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 373025 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 373044 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 373048 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 373064 T1143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@584439a9 name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 373064 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 373107 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 374110 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55152_h_dkw%2Fr
[junit4:junit4] 2> 374111 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55152_h_dkw%2Fr
[junit4:junit4] 2> 374114 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374115 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 374115 T1095 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374115 T1127 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374115 T1143 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374115 T1075 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374115 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374115 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374116 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374117 T1111 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374117 T1082 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 374122 T1144 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 374122 T1144 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 374377 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 374378 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54034/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:54034_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 374382 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55152/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:55152_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 374382 T1076 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 374382 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 374413 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374413 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374413 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374413 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374413 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374413 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375124 T1144 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/collection1
[junit4:junit4] 2> 375124 T1144 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 375125 T1144 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 375125 T1144 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 375127 T1144 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/collection1/'
[junit4:junit4] 2> 375127 T1144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/collection1/lib/README' to classloader
[junit4:junit4] 2> 375127 T1144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 375151 T1144 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 375184 T1144 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 375186 T1144 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 375191 T1144 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 375531 T1144 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 375531 T1144 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 375532 T1144 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 375538 T1144 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 375540 T1144 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 375555 T1144 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 375558 T1144 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 375560 T1144 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 375562 T1144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 375562 T1144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 375562 T1144 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 375563 T1144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 375563 T1144 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 375563 T1144 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 375563 T1144 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372002416009/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/
[junit4:junit4] 2> 375564 T1144 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 375564 T1144 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 375564 T1144 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4
[junit4:junit4] 2> 375564 T1144 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index/
[junit4:junit4] 2> 375565 T1144 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 375565 T1144 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index
[junit4:junit4] 2> 375568 T1144 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 375568 T1144 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 375569 T1144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 375570 T1144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 375570 T1144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 375570 T1144 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 375571 T1144 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 375571 T1144 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 375571 T1144 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 375571 T1144 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 375572 T1144 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 375573 T1144 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 375576 T1144 oass.SolrIndexSearcher.<init> Opening Searcher@16e46485 main
[junit4:junit4] 2> 375576 T1144 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 375577 T1144 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 375580 T1145 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16e46485 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 375581 T1144 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 375581 T1144 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55152/h_dkw/r collection:collection1 shard:shard1
[junit4:junit4] 2> 375585 T1144 oasc.ZkController.register We are http://127.0.0.1:55152/h_dkw/r/collection1/ and leader is http://127.0.0.1:34240/h_dkw/r/collection1/
[junit4:junit4] 2> 375585 T1144 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55152/h_dkw/r
[junit4:junit4] 2> 375585 T1144 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 375585 T1144 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2088 name=collection1 org.apache.solr.core.SolrCore@26d7abf0 url=http://127.0.0.1:55152/h_dkw/r/collection1 node=127.0.0.1:55152_h_dkw%2Fr C2088_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, base_url=http://127.0.0.1:55152/h_dkw/r, node_name=127.0.0.1:55152_h_dkw%2Fr, collection=collection1}
[junit4:junit4] 2> 375589 T1146 C2088 P55152 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 375589 T1144 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375589 T1146 C2088 P55152 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 375590 T1146 C2088 P55152 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 375590 T1146 C2088 P55152 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 375590 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 375591 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 375591 T1146 C2088 P55152 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 375591 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 375597 T1088 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 375677 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 375679 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47124
[junit4:junit4] 2> 375680 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 375680 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 375680 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724
[junit4:junit4] 2> 375681 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/solr.xml
[junit4:junit4] 2> 375681 T1053 oasc.CoreContainer.<init> New CoreContainer 645911672
[junit4:junit4] 2> 375681 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/'
[junit4:junit4] 2> 375681 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/'
[junit4:junit4] 2> 375751 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 375751 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 375752 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 375752 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 375753 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 375753 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 375753 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 375753 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 375754 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 375754 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 375759 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 375759 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 375760 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 375760 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 375762 T1158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c57d4e9 name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 375763 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 375764 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 375768 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 375771 T1160 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f3e1eed name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 375771 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 375775 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 375922 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375923 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55152/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:55152_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 375926 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375927 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375927 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375927 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375927 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375927 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375927 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376597 T1088 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 376598 T1088 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&core=collection1&version=2&action=PREPRECOVERY&onlyIfLeader=true&checkLive=true&wt=javabin&nodeName=127.0.0.1:55152_h_dkw%252Fr} status=0 QTime=1001
[junit4:junit4] 2> 376778 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47124_h_dkw%2Fr
[junit4:junit4] 2> 376779 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47124_h_dkw%2Fr
[junit4:junit4] 2> 376782 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376782 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376782 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376782 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376783 T1082 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376782 T1095 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376782 T1143 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376783 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 376783 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 376783 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 376784 T1127 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376784 T1111 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376785 T1075 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376786 T1160 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 376790 T1161 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 376790 T1161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 377432 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 377433 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47124/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:47124_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 377433 T1076 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 377434 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 377437 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377437 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377437 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377437 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377437 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377437 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377437 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 377791 T1161 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/collection1
[junit4:junit4] 2> 377792 T1161 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 377792 T1161 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 377792 T1161 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 377794 T1161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/collection1/'
[junit4:junit4] 2> 377794 T1161 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/collection1/lib/README' to classloader
[junit4:junit4] 2> 377794 T1161 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 377822 T1161 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 377850 T1161 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 377852 T1161 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 377856 T1161 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 378180 T1161 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 378180 T1161 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 378181 T1161 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 378190 T1161 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 378192 T1161 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 378205 T1161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 378208 T1161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 378211 T1161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 378212 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 378212 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 378212 T1161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 378213 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 378213 T1161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 378213 T1161 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 378213 T1161 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372002418724/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/
[junit4:junit4] 2> 378214 T1161 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 378214 T1161 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 378214 T1161 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5
[junit4:junit4] 2> 378215 T1161 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index/
[junit4:junit4] 2> 378215 T1161 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 378215 T1161 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index
[junit4:junit4] 2> 378218 T1161 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 378218 T1161 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 378220 T1161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 378220 T1161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 378220 T1161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 378221 T1161 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 378221 T1161 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 378221 T1161 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 378221 T1161 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 378222 T1161 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 378222 T1161 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 378224 T1161 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 378227 T1161 oass.SolrIndexSearcher.<init> Opening Searcher@3fc96394 main
[junit4:junit4] 2> 378227 T1161 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 378227 T1161 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 378230 T1162 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fc96394 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 378232 T1161 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 378232 T1161 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47124/h_dkw/r collection:collection1 shard:shard3
[junit4:junit4] 2> 378235 T1161 oasc.ZkController.register We are http://127.0.0.1:47124/h_dkw/r/collection1/ and leader is http://127.0.0.1:54594/h_dkw/r/collection1/
[junit4:junit4] 2> 378235 T1161 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47124/h_dkw/r
[junit4:junit4] 2> 378235 T1161 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 378235 T1161 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2089 name=collection1 org.apache.solr.core.SolrCore@549e81e0 url=http://127.0.0.1:47124/h_dkw/r/collection1 node=127.0.0.1:47124_h_dkw%2Fr C2089_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, base_url=http://127.0.0.1:47124/h_dkw/r, node_name=127.0.0.1:47124_h_dkw%2Fr, collection=collection1}
[junit4:junit4] 2> 378236 T1163 C2089 P47124 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 378236 T1161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 378236 T1163 C2089 P47124 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 378237 T1163 C2089 P47124 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 378237 T1163 C2089 P47124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 378237 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 378238 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 378238 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 378238 T1163 C2089 P47124 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 378243 T1106 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 378346 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 378349 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57492
[junit4:junit4] 2> 378350 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 378350 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 378350 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371
[junit4:junit4] 2> 378350 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/solr.xml
[junit4:junit4] 2> 378351 T1053 oasc.CoreContainer.<init> New CoreContainer 467102229
[junit4:junit4] 2> 378351 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/'
[junit4:junit4] 2> 378351 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/'
[junit4:junit4] 2> 378394 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 378395 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 378395 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 378395 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 378395 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 378396 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 378396 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 378396 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 378396 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 378397 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 378400 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 378401 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 378401 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 378402 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 378403 T1175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@102389f0 name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 378404 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 378406 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 378409 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 378411 T1177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e21aeb3 name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 378411 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 378415 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2090 name=collection1 org.apache.solr.core.SolrCore@26d7abf0 url=http://127.0.0.1:55152/h_dkw/r/collection1 node=127.0.0.1:55152_h_dkw%2Fr C2090_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, base_url=http://127.0.0.1:55152/h_dkw/r, node_name=127.0.0.1:55152_h_dkw%2Fr, shard=shard1, collection=collection1}
[junit4:junit4] 2> 378599 T1146 C2090 P55152 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:34240/h_dkw/r/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 378604 T1146 C2090 P55152 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 378611 T1146 C2090 P55152 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55152/h_dkw/r START replicas=[http://127.0.0.1:34240/h_dkw/r/collection1/] nUpdates=100
[junit4:junit4] 2> 378612 T1146 C2090 P55152 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 378613 T1146 C2090 P55152 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 378613 T1146 C2090 P55152 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 378613 T1146 C2090 P55152 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 378613 T1146 C2090 P55152 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 378614 T1146 C2090 P55152 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:34240/h_dkw/r/collection1/. core=collection1
[junit4:junit4] 2> 378614 T1146 C2090 P55152 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2091 name=collection1 org.apache.solr.core.SolrCore@7c7538c3 url=http://127.0.0.1:34240/h_dkw/r/collection1 node=127.0.0.1:34240_h_dkw%2Fr C2091_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, base_url=http://127.0.0.1:34240/h_dkw/r, node_name=127.0.0.1:34240_h_dkw%2Fr, shard=shard1, collection=collection1, leader=true}
[junit4:junit4] 2> 378618 T1090 C2091 P34240 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/get params={version=2&qt=/get&distrib=false&wt=javabin&getVersions=100} status=0 QTime=1
[junit4:junit4] 2> 378620 T1091 C2091 P34240 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 378622 T1091 C2091 P34240 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 378622 T1091 C2091 P34240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 378633 T1091 C2091 P34240 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 378633 T1091 C2091 P34240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 378634 T1091 C2091 P34240 oass.SolrIndexSearcher.<init> Opening Searcher@2c844761 realtime
[junit4:junit4] 2> 378634 T1091 C2091 P34240 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 378634 T1091 C2091 P34240 oasup.LogUpdateProcessor.finish [collection1] webapp=/h_dkw/r path=/update params={openSearcher=false&commit=true&version=2&waitSearcher=true&wt=javabin&softCommit=false&commit_end_point=true} {commit=} 0 14
[junit4:junit4] 2> 378655 T1146 C2090 P55152 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 378655 T1146 C2090 P55152 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> ASYNC NEW_CORE C2092 name=collection1 org.apache.solr.core.SolrCore@26d7abf0 url=http://127.0.0.1:55152/h_dkw/r/collection1 node=127.0.0.1:55152_h_dkw%2Fr C2092_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, base_url=http://127.0.0.1:55152/h_dkw/r, node_name=127.0.0.1:55152_h_dkw%2Fr, shard=shard1, collection=collection1}
[junit4:junit4] 2> 378680 T1146 C2092 P55152 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> ASYNC NEW_CORE C2093 name=collection1 org.apache.solr.core.SolrCore@7c7538c3 url=http://127.0.0.1:34240/h_dkw/r/collection1 node=127.0.0.1:34240_h_dkw%2Fr C2093_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, base_url=http://127.0.0.1:34240/h_dkw/r, node_name=127.0.0.1:34240_h_dkw%2Fr, shard=shard1, collection=collection1, leader=true}
[junit4:junit4] 2> 378684 T1088 C2093 P34240 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 378685 T1088 C2093 P34240 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/replication params={version=2&qt=/replication&wt=javabin&command=indexversion} status=0 QTime=4
[junit4:junit4] 2> 378686 T1146 C2092 P55152 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 378686 T1146 C2092 P55152 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 378686 T1146 C2092 P55152 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 378688 T1090 C2093 P34240 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/replication params={generation=2&version=2&qt=/replication&wt=javabin&command=filelist} status=0 QTime=0
[junit4:junit4] 2> 378689 T1146 C2092 P55152 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 378690 T1146 C2092 P55152 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index.20130623114701818
[junit4:junit4] 2> 378690 T1146 C2092 P55152 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index.20130623114701818 lockFactory=org.apache.lucene.store.NativeFSLockFactory@41b51cf0 fullCopy=false
[junit4:junit4] 2> 378696 T1091 C2093 P34240 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/replication params={file=segments_2&generation=2&qt=/replication&checksum=true&wt=filestream&command=filecontent} status=0 QTime=2
[junit4:junit4] 2> 378701 T1146 C2092 P55152 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 378714 T1146 C2092 P55152 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 378714 T1146 C2092 P55152 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 378716 T1146 C2092 P55152 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 378717 T1146 C2092 P55152 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 378717 T1146 C2092 P55152 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 378717 T1146 C2092 P55152 oass.SolrIndexSearcher.<init> Opening Searcher@13c95e4c main
[junit4:junit4] 2> 378718 T1145 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13c95e4c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 378719 T1146 C2092 P55152 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index.20130623114701818 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index.20130623114701818;done=true>>]
[junit4:junit4] 2> 378719 T1146 C2092 P55152 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index.20130623114701818
[junit4:junit4] 2> 378719 T1146 C2092 P55152 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty4/index.20130623114701818
[junit4:junit4] 2> 378720 T1146 C2092 P55152 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 378720 T1146 C2092 P55152 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 378720 T1146 C2092 P55152 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 378720 T1146 C2092 P55152 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 378721 T1146 C2092 P55152 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 378942 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 378943 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47124/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:47124_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 378947 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55152/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:55152_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 378951 T1177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378951 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378951 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378952 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378951 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378952 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378951 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 378951 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379244 T1106 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 379244 T1106 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&core=collection1&version=2&action=PREPRECOVERY&onlyIfLeader=true&checkLive=true&wt=javabin&nodeName=127.0.0.1:47124_h_dkw%252Fr} status=0 QTime=1001
[junit4:junit4] 2> 379417 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57492_h_dkw%2Fr
[junit4:junit4] 2> 379419 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57492_h_dkw%2Fr
[junit4:junit4] 2> 379421 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379421 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379421 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379421 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379421 T1177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379421 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 379422 T1160 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379422 T1111 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379422 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 379422 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 379423 T1082 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379424 T1143 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379425 T1127 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379425 T1177 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379425 T1095 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379425 T1075 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 379432 T1180 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 379432 T1180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 380458 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 380459 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57492/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:57492_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 380459 T1076 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 380460 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 380463 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380463 T1177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380463 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380463 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380463 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380463 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380463 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 380464 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> ASYNC NEW_CORE C2094 name=collection1 org.apache.solr.core.SolrCore@549e81e0 url=http://127.0.0.1:47124/h_dkw/r/collection1 node=127.0.0.1:47124_h_dkw%2Fr C2094_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, base_url=http://127.0.0.1:47124/h_dkw/r, node_name=127.0.0.1:47124_h_dkw%2Fr, shard=shard3, collection=collection1}
[junit4:junit4] 2> 381245 T1163 C2094 P47124 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54594/h_dkw/r/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 381245 T1163 C2094 P47124 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47124/h_dkw/r START replicas=[http://127.0.0.1:54594/h_dkw/r/collection1/] nUpdates=100
[junit4:junit4] 2> 381247 T1163 C2094 P47124 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 381247 T1163 C2094 P47124 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 381247 T1163 C2094 P47124 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 381247 T1163 C2094 P47124 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 381248 T1163 C2094 P47124 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 381248 T1163 C2094 P47124 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54594/h_dkw/r/collection1/. core=collection1
[junit4:junit4] 2> 381248 T1163 C2094 P47124 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2095 name=collection1 org.apache.solr.core.SolrCore@2059bad8 url=http://127.0.0.1:54594/h_dkw/r/collection1 node=127.0.0.1:54594_h_dkw%2Fr C2095_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, base_url=http://127.0.0.1:54594/h_dkw/r, node_name=127.0.0.1:54594_h_dkw%2Fr, shard=shard3, collection=collection1, leader=true}
[junit4:junit4] 2> 381254 T1106 C2095 P54594 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/get params={version=2&qt=/get&distrib=false&wt=javabin&getVersions=100} status=0 QTime=1
[junit4:junit4] 2> 381254 T1107 C2095 P54594 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 381257 T1107 C2095 P54594 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 381257 T1107 C2095 P54594 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 381263 T1107 C2095 P54594 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 381263 T1107 C2095 P54594 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 381264 T1107 C2095 P54594 oass.SolrIndexSearcher.<init> Opening Searcher@2f51fc24 realtime
[junit4:junit4] 2> 381264 T1107 C2095 P54594 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 381264 T1107 C2095 P54594 oasup.LogUpdateProcessor.finish [collection1] webapp=/h_dkw/r path=/update params={openSearcher=false&commit=true&version=2&waitSearcher=true&wt=javabin&softCommit=false&commit_end_point=true} {commit=} 0 11
[junit4:junit4] 2> 381265 T1163 C2094 P47124 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 381265 T1163 C2094 P47124 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 381268 T1105 C2095 P54594 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 381268 T1105 C2095 P54594 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/replication params={version=2&qt=/replication&wt=javabin&command=indexversion} status=0 QTime=1
[junit4:junit4] 2> 381268 T1163 C2094 P47124 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 381269 T1163 C2094 P47124 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 381269 T1163 C2094 P47124 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 381271 T1106 C2095 P54594 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/replication params={generation=2&version=2&qt=/replication&wt=javabin&command=filelist} status=0 QTime=1
[junit4:junit4] 2> 381271 T1163 C2094 P47124 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 381272 T1163 C2094 P47124 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index.20130623114704400
[junit4:junit4] 2> 381272 T1163 C2094 P47124 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index.20130623114704400 lockFactory=org.apache.lucene.store.NativeFSLockFactory@35b2696d fullCopy=false
[junit4:junit4] 2> 381274 T1107 C2095 P54594 oasc.SolrCore.execute [collection1] webapp=/h_dkw/r path=/replication params={file=segments_2&generation=2&qt=/replication&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4:junit4] 2> 381275 T1163 C2094 P47124 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 381283 T1163 C2094 P47124 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 381283 T1163 C2094 P47124 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 381285 T1163 C2094 P47124 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 381286 T1163 C2094 P47124 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 381286 T1163 C2094 P47124 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 381287 T1163 C2094 P47124 oass.SolrIndexSearcher.<init> Opening Searcher@19a94d3f main
[junit4:junit4] 2> 381287 T1162 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19a94d3f main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 381288 T1163 C2094 P47124 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index.20130623114704400 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index.20130623114704400;done=true>>]
[junit4:junit4] 2> 381288 T1163 C2094 P47124 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index.20130623114704400
[junit4:junit4] 2> 381288 T1163 C2094 P47124 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty5/index.20130623114704400
[junit4:junit4] 2> 381289 T1163 C2094 P47124 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 381289 T1163 C2094 P47124 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 381289 T1163 C2094 P47124 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 381289 T1163 C2094 P47124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 381291 T1163 C2094 P47124 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 381434 T1180 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/collection1
[junit4:junit4] 2> 381434 T1180 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 381435 T1180 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 381435 T1180 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 381437 T1180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/collection1/'
[junit4:junit4] 2> 381437 T1180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/collection1/lib/README' to classloader
[junit4:junit4] 2> 381438 T1180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 381470 T1180 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 381505 T1180 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 381507 T1180 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 381511 T1180 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 381869 T1180 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 381869 T1180 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 381870 T1180 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 381875 T1180 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 381877 T1180 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 381893 T1180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 381897 T1180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 381900 T1180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 381901 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 381901 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 381901 T1180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 381902 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 381903 T1180 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 381903 T1180 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 381903 T1180 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372002421371/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty6/
[junit4:junit4] 2> 381903 T1180 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 381904 T1180 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 381904 T1180 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty6
[junit4:junit4] 2> 381904 T1180 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty6/index/
[junit4:junit4] 2> 381905 T1180 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 381905 T1180 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty6/index
[junit4:junit4] 2> 381924 T1180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 381924 T1180 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 381926 T1180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 381926 T1180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 381927 T1180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 381927 T1180 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 381928 T1180 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 381928 T1180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 381930 T1180 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 381931 T1180 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 381931 T1180 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 381933 T1180 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 381935 T1180 oass.SolrIndexSearcher.<init> Opening Searcher@172ca57d main
[junit4:junit4] 2> 381936 T1180 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 381936 T1180 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 381940 T1183 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@172ca57d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 381941 T1180 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 381941 T1180 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57492/h_dkw/r collection:collection1 shard:shard2
[junit4:junit4] 2> 381945 T1180 oasc.ZkController.register We are http://127.0.0.1:57492/h_dkw/r/collection1/ and leader is http://127.0.0.1:54034/h_dkw/r/collection1/
[junit4:junit4] 2> 381945 T1180 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57492/h_dkw/r
[junit4:junit4] 2> 381945 T1180 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 381945 T1180 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2096 name=collection1 org.apache.solr.core.SolrCore@7d5a4e60 url=http://127.0.0.1:57492/h_dkw/r/collection1 node=127.0.0.1:57492_h_dkw%2Fr C2096_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, base_url=http://127.0.0.1:57492/h_dkw/r, node_name=127.0.0.1:57492_h_dkw%2Fr, collection=collection1}
[junit4:junit4] 2> 381945 T1184 C2096 P57492 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 381946 T1184 C2096 P57492 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 381946 T1184 C2096 P57492 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 381946 T1180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 381946 T1184 C2096 P57492 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 381947 T1053 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 381947 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 381947 T1184 C2096 P57492 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 381947 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 381954 T1121 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 381969 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 381970 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47124/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:47124_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 381973 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57492/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:57492_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 381978 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 381978 T1177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 382028 T1053 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 382031 T1053 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43809
[junit4:junit4] 2> 382031 T1053 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 382031 T1053 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 382031 T1053 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082
[junit4:junit4] 2> 382032 T1053 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/solr.xml
[junit4:junit4] 2> 382032 T1053 oasc.CoreContainer.<init> New CoreContainer 1189707094
[junit4:junit4] 2> 382032 T1053 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/'
[junit4:junit4] 2> 382032 T1053 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/'
[junit4:junit4] 2> 382076 T1053 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 382076 T1053 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 382076 T1053 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 382077 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 382077 T1053 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 382077 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 382077 T1053 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 382077 T1053 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 382078 T1053 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 382078 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 382083 T1053 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 382083 T1053 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52305/solr
[junit4:junit4] 2> 382083 T1053 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 382084 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 382086 T1196 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1febc1d3 name:ZooKeeperConnection Watcher:127.0.0.1:52305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 382086 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 382088 T1053 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 382092 T1053 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 382093 T1198 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37d5e4e0 name:ZooKeeperConnection Watcher:127.0.0.1:52305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 382094 T1053 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 382098 T1053 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 382954 T1121 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 382955 T1121 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&core=collection1&version=2&action=PREPRECOVERY&onlyIfLeader=true&checkLive=true&wt=javabin&nodeName=127.0.0.1:57492_h_dkw%252Fr} status=0 QTime=1001
[junit4:junit4] 2> 383101 T1053 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43809_h_dkw%2Fr
[junit4:junit4] 2> 383104 T1053 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43809_h_dkw%2Fr
[junit4:junit4] 2> 383107 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 383108 T1177 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383108 T1082 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383108 T1143 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383108 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383108 T1127 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383108 T1111 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383108 T1160 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383109 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383108 T1095 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383108 T1177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383109 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383109 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383109 T1198 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383109 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383109 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383110 T1075 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 383116 T1199 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 383117 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 383488 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 383489 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43809/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:43809_h_dkw%2Fr",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 383490 T1076 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 383490 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 383494 T1111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1095 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 383494 T1177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 384118 T1199 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/collection1
[junit4:junit4] 2> 384118 T1199 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 384119 T1199 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 384119 T1199 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 384121 T1199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/collection1/'
[junit4:junit4] 2> 384121 T1199 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/collection1/lib/README' to classloader
[junit4:junit4] 2> 384121 T1199 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 384152 T1199 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 384187 T1199 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 384189 T1199 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 384193 T1199 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 384518 T1199 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 384518 T1199 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 384519 T1199 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 384524 T1199 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 384526 T1199 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 384544 T1199 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 384548 T1199 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 384551 T1199 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 384553 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 384553 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 384553 T1199 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 384554 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 384554 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 384555 T1199 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 384555 T1199 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372002425082/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty7/
[junit4:junit4] 2> 384555 T1199 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d6ebabd
[junit4:junit4] 2> 384555 T1199 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 384556 T1199 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty7
[junit4:junit4] 2> 384556 T1199 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty7/index/
[junit4:junit4] 2> 384556 T1199 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 384557 T1199 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty7/index
[junit4:junit4] 2> 384559 T1199 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 384559 T1199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 384561 T1199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 384561 T1199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 384562 T1199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into upda
[...truncated too long message...]
127.0.0.1:34815/h_dkw/r",
[junit4:junit4] 2> "node_name":"127.0.0.1:34815_h_dkw%2Fr",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "operation":"state"}
[junit4:junit4] 2> 498371 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 498376 T1053 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/h_dkw/r,null}
[junit4:junit4] 2> 498427 T1053 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53431
[junit4:junit4] 2> 498427 T1053 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=308909543
[junit4:junit4] 2> 499429 T1053 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 499429 T1053 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 499431 T1053 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@26ec059d
[junit4:junit4] 2> 499438 T1053 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=45,cumulative_deletesById=19,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 499439 T1053 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 499439 T1053 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 499439 T1053 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> ASYNC NEW_CORE C2189 name=collection1 org.apache.solr.core.SolrCore@26ec059d url=http://127.0.0.1:53431/h_dkw/r/collection1 node=127.0.0.1:53431_h_dkw%2Fr C2189_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, base_url=http://127.0.0.1:53431/h_dkw/r, node_name=127.0.0.1:53431_h_dkw%2Fr, shard=shard2, collection=collection1, leader=true}
[junit4:junit4] 2> 499454 T1053 C2189 P53431 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12/index,segFN=segments_q,generation=26}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12/index,segFN=segments_r,generation=27}
[junit4:junit4] 2> 499455 T1053 C2189 P53431 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 27
[junit4:junit4] 2> 499456 T1053 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 499457 T1053 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 499457 T1053 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12;done=false>>]
[junit4:junit4] 2> 499457 T1053 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12
[junit4:junit4] 2> 499457 T1053 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12/index;done=false>>]
[junit4:junit4] 2> 499457 T1053 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372002403441/jetty12/index
[junit4:junit4] 2> 499458 T1551 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89915549512040476-127.0.0.1:53431_h_dkw%2Fr-n_0000000012) am no longer a leader.
[junit4:junit4] 2> 499460 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 499461 T1291 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 499461 T1291 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 499482 T1053 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/h_dkw/r,null}
[junit4:junit4] 2> 499554 T1053 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 499556 T1053 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52305 52305
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=502D08402E749A31 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ca_ES -Dtests.timezone=America/Nassau -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 139s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard2 is not consistent. Got 26 from http://127.0.0.1:57492/h_dkw/r/collection1lastClient and got 25 from http://127.0.0.1:53431/h_dkw/r/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([502D08402E749A31:D1CB8658592BFA0D]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 499587 T1053 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 139284 T1052 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 499655 T1274 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 499655 T1274 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 499655 T1274 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=ca_ES, timezone=America/Nassau
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=2,free=120145384,total=218472448
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestDFRSimilarityFactory, TestNumberUtils, ClusterStateTest, SampleTest, TestRandomFaceting, HighlighterConfigTest, TestUpdate, TestSolrJ, TestMultiCoreConfBootstrap, TestSolr4Spatial, SuggesterWFSTTest, TestManagedSchema, TestSerializedLuceneMatchVersion, BasicZkTest, BasicDistributedZk2Test, TestWordDelimiterFilterFactory, EchoParamsTest, BadIndexSchemaTest, TestSchemaNameResource, TestFieldTypeResource, SignatureUpdateProcessorFactoryTest, HighlighterTest, TestDistributedGrouping, TestDistributedSearch, UpdateParamsTest, TestStressReorder, LeaderElectionIntegrationTest, CoreContainerCoreInitFailuresTest, IndexBasedSpellCheckerTest, TestSuggestSpellingConverter, SimplePostToolTest, NotRequiredUniqueKeyTest, StandardRequestHandlerTest, SpellPossibilityIteratorTest, DocValuesMultiTest, TestOmitPositions, OutputWriterTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 139.81s, 1 test, 1 failure <<< FAILURES!
[...truncated 576 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1256 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 38 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b93 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 6203
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6203/
Java: 32bit/jdk1.6.0_45 -client -XX:+UseParallelGC
3 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<200> but was:<50>
Stack Trace:
java.lang.AssertionError: expected:<200> but was:<50>
at __randomizedtesting.SeedInfo.seed([AB67D61ACD08DBAF:2A815802BA57BB93]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:33034/collection1lastClient and got 262 from http://127.0.0.1:38345/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:33034/collection1lastClient and got 262 from http://127.0.0.1:38345/collection1
at __randomizedtesting.SeedInfo.seed([AB67D61ACD08DBAF:2A815802BA57BB93]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
REGRESSION: org.apache.solr.handler.TestReplicationHandler.doTestStressReplication
Error Message:
timed out waiting for collection1 startAt time to exceed: Mon Jun 24 04:13:53 PONT 2013
Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Mon Jun 24 04:13:53 PONT 2013
at __randomizedtesting.SeedInfo.seed([AB67D61ACD08DBAF:70CCD6DCC820B21C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1473)
at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:777)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Build Log:
[...truncated 9044 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 728645 T1831 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 728648 T1831 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1372007488197
[junit4:junit4] 2> 728649 T1831 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 728649 T1832 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 728749 T1831 oasc.ZkTestServer.run start zk server on port:51185
[junit4:junit4] 2> 728750 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 728754 T1838 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d9b109 name:ZooKeeperConnection Watcher:127.0.0.1:51185 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 728754 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 728754 T1831 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 728758 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 728759 T1840 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36d62a name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 728759 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 728759 T1831 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 728762 T1831 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 728764 T1831 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 728765 T1831 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 728767 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 728767 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 728770 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 728770 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 728773 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 728773 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 728775 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 728776 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 728777 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 728778 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 728780 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 728781 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 728783 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 728783 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 728785 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 728786 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 728787 T1831 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 728788 T1831 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 728857 T1831 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 728860 T1831 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49240
[junit4:junit4] 2> 728861 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 728861 T1831 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 728861 T1831 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341
[junit4:junit4] 2> 728862 T1831 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/solr.xml
[junit4:junit4] 2> 728862 T1831 oasc.CoreContainer.<init> New CoreContainer 27051290
[junit4:junit4] 2> 728862 T1831 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/'
[junit4:junit4] 2> 728863 T1831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/'
[junit4:junit4] 2> 728923 T1831 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 728924 T1831 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 728924 T1831 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 728924 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 728925 T1831 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 728925 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 728926 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 728926 T1831 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 728926 T1831 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 728927 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 728930 T1831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 728930 T1831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51185/solr
[junit4:junit4] 2> 728930 T1831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 728931 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 728933 T1851 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1420e33 name:ZooKeeperConnection Watcher:127.0.0.1:51185 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 728933 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 728935 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 728938 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 728939 T1853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11d96e5 name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 728939 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 728940 T1831 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 728942 T1831 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 728944 T1831 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 728945 T1831 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49240_
[junit4:junit4] 2> 728946 T1831 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49240_
[junit4:junit4] 2> 728948 T1831 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 728952 T1831 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 728953 T1831 oasc.Overseer.start Overseer (id=89915882746609667-127.0.0.1:49240_-n_0000000000) starting
[junit4:junit4] 2> 728955 T1831 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 728959 T1855 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 728959 T1831 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 728962 T1831 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 728963 T1831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 728965 T1854 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 728967 T1856 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 728967 T1856 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 730467 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 730468 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49240_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49240"}
[junit4:junit4] 2> 730468 T1854 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 730469 T1854 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 730472 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 730968 T1856 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/collection1
[junit4:junit4] 2> 730969 T1856 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 730970 T1856 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 730970 T1856 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 730971 T1856 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/collection1/'
[junit4:junit4] 2> 730972 T1856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/collection1/lib/README' to classloader
[junit4:junit4] 2> 730973 T1856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 731024 T1856 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 731104 T1856 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 731107 T1856 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 731123 T1856 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 731993 T1856 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 731994 T1856 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 731995 T1856 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 732005 T1856 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 732010 T1856 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 732032 T1856 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 732053 T1856 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 732058 T1856 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 732059 T1856 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 732059 T1856 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 732060 T1856 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 732061 T1856 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 732061 T1856 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 732062 T1856 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 732072 T1856 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372007488341/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/
[junit4:junit4] 2> 732073 T1856 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497062
[junit4:junit4] 2> 732073 T1856 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 732074 T1856 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data
[junit4:junit4] 2> 732074 T1856 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index/
[junit4:junit4] 2> 732075 T1856 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 732076 T1856 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index
[junit4:junit4] 2> 732089 T1856 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6899a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 732090 T1856 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 732092 T1856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 732092 T1856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 732092 T1856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 732093 T1856 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 732094 T1856 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 732094 T1856 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 732094 T1856 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 732095 T1856 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 732095 T1856 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 732098 T1856 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 732103 T1856 oass.SolrIndexSearcher.<init> Opening Searcher@1206dd4 main
[junit4:junit4] 2> 732104 T1856 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 732105 T1856 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 732109 T1857 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1206dd4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 732111 T1856 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 732111 T1856 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49240 collection:control_collection shard:shard1
[junit4:junit4] 2> 732112 T1856 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 732119 T1856 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 732122 T1856 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 732122 T1856 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 732122 T1856 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49240/collection1/
[junit4:junit4] 2> 732123 T1856 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 732123 T1856 oasc.SyncStrategy.syncToMe http://127.0.0.1:49240/collection1/ has no replicas
[junit4:junit4] 2> 732123 T1856 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49240/collection1/
[junit4:junit4] 2> 732124 T1856 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 733477 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 733484 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 733532 T1856 oasc.ZkController.register We are http://127.0.0.1:49240/collection1/ and leader is http://127.0.0.1:49240/collection1/
[junit4:junit4] 2> 733532 T1856 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49240
[junit4:junit4] 2> 733532 T1856 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 733533 T1856 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 733533 T1856 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 733535 T1856 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 733536 T1831 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 733536 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 733537 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 733540 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 733541 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 733542 T1860 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e0cae6 name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 733542 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 733544 T1831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 733546 T1831 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 733614 T1831 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 733616 T1831 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33034
[junit4:junit4] 2> 733617 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 733617 T1831 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 733618 T1831 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094
[junit4:junit4] 2> 733618 T1831 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/solr.xml
[junit4:junit4] 2> 733618 T1831 oasc.CoreContainer.<init> New CoreContainer 24425879
[junit4:junit4] 2> 733619 T1831 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/'
[junit4:junit4] 2> 733619 T1831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/'
[junit4:junit4] 2> 733671 T1831 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 733671 T1831 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 733672 T1831 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 733672 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 733672 T1831 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 733673 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 733673 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 733673 T1831 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 733674 T1831 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 733674 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 733677 T1831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 733678 T1831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51185/solr
[junit4:junit4] 2> 733678 T1831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 733679 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 733680 T1871 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f4e17c name:ZooKeeperConnection Watcher:127.0.0.1:51185 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 733680 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 733682 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 733684 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 733692 T1873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@117b032 name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 733692 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 733696 T1831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 734698 T1831 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33034_
[junit4:junit4] 2> 734699 T1831 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33034_
[junit4:junit4] 2> 734702 T1853 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 734702 T1873 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 734702 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 734702 T1860 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 734707 T1874 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 734707 T1874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 734989 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 734989 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49240__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49240_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49240"}
[junit4:junit4] 2> 734992 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33034_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33034"}
[junit4:junit4] 2> 734992 T1854 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 734992 T1854 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 734995 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 734995 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 734995 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 735708 T1874 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/collection1
[junit4:junit4] 2> 735708 T1874 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 735709 T1874 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 735709 T1874 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 735711 T1874 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/collection1/'
[junit4:junit4] 2> 735711 T1874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/collection1/lib/README' to classloader
[junit4:junit4] 2> 735711 T1874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 735764 T1874 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 735804 T1874 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 735806 T1874 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 735810 T1874 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 736266 T1874 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 736266 T1874 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 736267 T1874 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 736274 T1874 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 736277 T1874 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 736287 T1874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 736290 T1874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 736293 T1874 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 736294 T1874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 736294 T1874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 736295 T1874 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 736296 T1874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 736296 T1874 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 736296 T1874 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 736296 T1874 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372007493094/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/
[junit4:junit4] 2> 736297 T1874 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497062
[junit4:junit4] 2> 736297 T1874 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 736297 T1874 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1
[junit4:junit4] 2> 736298 T1874 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index/
[junit4:junit4] 2> 736298 T1874 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 736298 T1874 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index
[junit4:junit4] 2> 736301 T1874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 736302 T1874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 736303 T1874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 736304 T1874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 736304 T1874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 736305 T1874 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 736305 T1874 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 736306 T1874 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 736306 T1874 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 736306 T1874 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 736307 T1874 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 736309 T1874 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 736312 T1874 oass.SolrIndexSearcher.<init> Opening Searcher@174a00c main
[junit4:junit4] 2> 736313 T1874 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 736313 T1874 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 736344 T1875 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@174a00c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 736346 T1874 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 736346 T1874 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33034 collection:collection1 shard:shard1
[junit4:junit4] 2> 736347 T1874 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 736351 T1874 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 736353 T1874 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 736353 T1874 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 736353 T1874 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 736353 T1874 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 736354 T1874 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/ has no replicas
[junit4:junit4] 2> 736354 T1874 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 736354 T1874 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 736500 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 736506 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 736506 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 736506 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 736508 T1874 oasc.ZkController.register We are http://127.0.0.1:33034/collection1/ and leader is http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 736509 T1874 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33034
[junit4:junit4] 2> 736509 T1874 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 736509 T1874 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 736510 T1874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 736511 T1874 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 736512 T1831 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 736512 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 736512 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 736583 T1831 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 736585 T1831 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46648
[junit4:junit4] 2> 736586 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 736586 T1831 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 736587 T1831 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063
[junit4:junit4] 2> 736587 T1831 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/solr.xml
[junit4:junit4] 2> 736588 T1831 oasc.CoreContainer.<init> New CoreContainer 1498189
[junit4:junit4] 2> 736588 T1831 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/'
[junit4:junit4] 2> 736588 T1831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/'
[junit4:junit4] 2> 736639 T1831 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 736640 T1831 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 736640 T1831 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 736640 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 736641 T1831 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 736641 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 736641 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 736642 T1831 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 736642 T1831 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 736643 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 736645 T1831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 736646 T1831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51185/solr
[junit4:junit4] 2> 736646 T1831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 736647 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 736649 T1887 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48bff1 name:ZooKeeperConnection Watcher:127.0.0.1:51185 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 736649 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 736652 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 736654 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 736655 T1889 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e95b36 name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 736656 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 736659 T1831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 737661 T1831 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46648_
[junit4:junit4] 2> 737662 T1831 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46648_
[junit4:junit4] 2> 737665 T1860 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 737666 T1873 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 737666 T1889 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 737666 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 737666 T1853 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 737666 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 737667 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 737673 T1890 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 737673 T1890 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 738012 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 738013 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:33034__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33034_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33034"}
[junit4:junit4] 2> 738017 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46648_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46648"}
[junit4:junit4] 2> 738017 T1854 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 738017 T1854 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 738022 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 738022 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 738022 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 738022 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 738674 T1890 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/collection1
[junit4:junit4] 2> 738675 T1890 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 738675 T1890 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 738676 T1890 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 738677 T1890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/collection1/'
[junit4:junit4] 2> 738678 T1890 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/collection1/lib/README' to classloader
[junit4:junit4] 2> 738678 T1890 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 738746 T1890 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 738813 T1890 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 738816 T1890 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 738823 T1890 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 739493 T1890 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 739493 T1890 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 739494 T1890 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 739502 T1890 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 739507 T1890 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 739523 T1890 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 739528 T1890 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 739533 T1890 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 739534 T1890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 739535 T1890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 739535 T1890 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 739536 T1890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 739536 T1890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 739537 T1890 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 739537 T1890 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372007496063/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/
[junit4:junit4] 2> 739537 T1890 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497062
[junit4:junit4] 2> 739538 T1890 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 739539 T1890 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2
[junit4:junit4] 2> 739539 T1890 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index/
[junit4:junit4] 2> 739540 T1890 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 739540 T1890 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index
[junit4:junit4] 2> 739544 T1890 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 739544 T1890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 739547 T1890 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 739547 T1890 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 739548 T1890 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 739549 T1890 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 739550 T1890 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 739550 T1890 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 739551 T1890 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 739552 T1890 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 739552 T1890 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 739555 T1890 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 739560 T1890 oass.SolrIndexSearcher.<init> Opening Searcher@4a7f25 main
[junit4:junit4] 2> 739561 T1890 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 739561 T1890 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 739568 T1891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a7f25 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 739570 T1890 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 739571 T1890 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46648 collection:collection1 shard:shard1
[junit4:junit4] 2> 739573 T1890 oasc.ZkController.register We are http://127.0.0.1:46648/collection1/ and leader is http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 739574 T1890 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46648
[junit4:junit4] 2> 739574 T1890 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 739574 T1890 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C470 name=collection1 org.apache.solr.core.SolrCore@89f479 url=http://127.0.0.1:46648/collection1 node=127.0.0.1:46648_ C470_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:46648_, base_url=http://127.0.0.1:46648}
[junit4:junit4] 2> 739575 T1892 C470 P46648 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 739575 T1890 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 739576 T1892 C470 P46648 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 739577 T1892 C470 P46648 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 739577 T1892 C470 P46648 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 739577 T1831 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 739578 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 739578 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 739593 T1892 C470 P46648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 739598 T1868 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:46648__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 739684 T1831 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 739702 T1831 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38345
[junit4:junit4] 2> 739703 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 739703 T1831 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 739704 T1831 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131
[junit4:junit4] 2> 739704 T1831 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/solr.xml
[junit4:junit4] 2> 739705 T1831 oasc.CoreContainer.<init> New CoreContainer 5656446
[junit4:junit4] 2> 739705 T1831 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/'
[junit4:junit4] 2> 739706 T1831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/'
[junit4:junit4] 2> 739824 T1831 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 739825 T1831 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 739826 T1831 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 739826 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 739827 T1831 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 739827 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 739828 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 739828 T1831 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 739829 T1831 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 739830 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 739833 T1831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 739834 T1831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51185/solr
[junit4:junit4] 2> 739835 T1831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 739845 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 739846 T1904 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@84d2fc name:ZooKeeperConnection Watcher:127.0.0.1:51185 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 739851 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 739853 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 739858 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 739859 T1906 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a20982 name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 739859 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 739863 T1831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 740868 T1831 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38345_
[junit4:junit4] 2> 740869 T1831 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38345_
[junit4:junit4] 2> 740872 T1853 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 740872 T1889 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 740872 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 740872 T1906 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 740872 T1873 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 740872 T1860 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 740873 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 740872 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 740873 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 740877 T1907 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 740878 T1907 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 741028 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 741029 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46648__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46648_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46648"}
[junit4:junit4] 2> 741032 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38345_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38345"}
[junit4:junit4] 2> 741032 T1854 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 741032 T1854 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 741035 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 741035 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 741035 T1906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 741035 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 741035 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 741599 T1868 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:46648__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 741599 T1868 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:46648__collection1&state=recovering&nodeName=127.0.0.1:46648_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001
[junit4:junit4] 2> 741879 T1907 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/collection1
[junit4:junit4] 2> 741880 T1907 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 741881 T1907 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 741881 T1907 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 741882 T1907 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/collection1/'
[junit4:junit4] 2> 741882 T1907 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/collection1/lib/README' to classloader
[junit4:junit4] 2> 741883 T1907 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 741913 T1907 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 741962 T1907 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 741964 T1907 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 741969 T1907 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 742395 T1907 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 742396 T1907 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 742397 T1907 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 742403 T1907 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 742406 T1907 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 742416 T1907 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 742419 T1907 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 742422 T1907 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 742423 T1907 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 742423 T1907 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 742423 T1907 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 742424 T1907 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 742424 T1907 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 742425 T1907 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 742425 T1907 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372007499131/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/
[junit4:junit4] 2> 742425 T1907 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497062
[junit4:junit4] 2> 742425 T1907 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 742426 T1907 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3
[junit4:junit4] 2> 742426 T1907 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index/
[junit4:junit4] 2> 742426 T1907 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 742427 T1907 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index
[junit4:junit4] 2> 742430 T1907 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 742430 T1907 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 742431 T1907 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 742432 T1907 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 742432 T1907 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 742433 T1907 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 742433 T1907 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 742434 T1907 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 742434 T1907 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 742435 T1907 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 742435 T1907 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 742437 T1907 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 742440 T1907 oass.SolrIndexSearcher.<init> Opening Searcher@fdefa3 main
[junit4:junit4] 2> 742440 T1907 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 742441 T1907 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 742446 T1908 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fdefa3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 742447 T1907 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 742448 T1907 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38345 collection:collection1 shard:shard1
[junit4:junit4] 2> 742451 T1907 oasc.ZkController.register We are http://127.0.0.1:38345/collection1/ and leader is http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 742451 T1907 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38345
[junit4:junit4] 2> 742451 T1907 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 742451 T1907 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C471 name=collection1 org.apache.solr.core.SolrCore@18bb8a url=http://127.0.0.1:38345/collection1 node=127.0.0.1:38345_ C471_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:38345_, base_url=http://127.0.0.1:38345}
[junit4:junit4] 2> 742452 T1909 C471 P38345 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 742452 T1907 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 742452 T1909 C471 P38345 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 742453 T1909 C471 P38345 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 742453 T1909 C471 P38345 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 742453 T1831 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 742453 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 742454 T1909 C471 P38345 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 742454 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 742457 T1869 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:38345__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 742522 T1831 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 742524 T1831 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48600
[junit4:junit4] 2> 742525 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 742525 T1831 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 742526 T1831 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005
[junit4:junit4] 2> 742526 T1831 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/solr.xml
[junit4:junit4] 2> 742526 T1831 oasc.CoreContainer.<init> New CoreContainer 21774103
[junit4:junit4] 2> 742527 T1831 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/'
[junit4:junit4] 2> 742527 T1831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/'
[junit4:junit4] 2> 742578 T1831 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 742578 T1831 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 742579 T1831 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 742579 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 742579 T1831 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 742580 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 742580 T1831 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 742580 T1831 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 742581 T1831 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 742581 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 742584 T1831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 742584 T1831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51185/solr
[junit4:junit4] 2> 742585 T1831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 742586 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 742587 T1921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@265e80 name:ZooKeeperConnection Watcher:127.0.0.1:51185 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 742587 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 742589 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 742592 T1831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 742593 T1923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@814915 name:ZooKeeperConnection Watcher:127.0.0.1:51185/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 742593 T1831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 742596 T1831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 742622 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 742623 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38345__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38345_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38345"}
[junit4:junit4] 2> 742625 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 742625 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 742625 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 742625 T1906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 742625 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 742625 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 743458 T1869 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:38345__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 743458 T1869 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:38345__collection1&state=recovering&nodeName=127.0.0.1:38345_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 743599 T1831 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48600_
[junit4:junit4] 2> ASYNC NEW_CORE C472 name=collection1 org.apache.solr.core.SolrCore@89f479 url=http://127.0.0.1:46648/collection1 node=127.0.0.1:46648_ C472_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:46648_, base_url=http://127.0.0.1:46648}
[junit4:junit4] 2> 743600 T1892 C472 P46648 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33034/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 743600 T1831 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48600_
[junit4:junit4] 2> 743601 T1892 C472 P46648 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46648 START replicas=[http://127.0.0.1:33034/collection1/] nUpdates=100
[junit4:junit4] 2> 743601 T1892 C472 P46648 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 743602 T1892 C472 P46648 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 743602 T1892 C472 P46648 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 743602 T1892 C472 P46648 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 743603 T1892 C472 P46648 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 743603 T1892 C472 P46648 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33034/collection1/. core=collection1
[junit4:junit4] 2> 743603 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 743603 T1892 C472 P46648 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 743605 T1873 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 743605 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 743606 T1860 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 743606 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 743606 T1906 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 743607 T1906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 743607 T1889 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 743607 T1853 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 743607 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 743607 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C473 name=collection1 org.apache.solr.core.SolrCore@537fb2 url=http://127.0.0.1:33034/collection1 node=127.0.0.1:33034_ C473_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:33034_, base_url=http://127.0.0.1:33034, leader=true}
[junit4:junit4] 2> 743610 T1869 C473 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 743610 T1923 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 743612 T1868 C473 P33034 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 743614 T1868 C473 P33034 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 743615 T1868 C473 P33034 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 743616 T1925 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 743616 T1925 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 743628 T1868 C473 P33034 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 743628 T1868 C473 P33034 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 743629 T1868 C473 P33034 oass.SolrIndexSearcher.<init> Opening Searcher@753f4a realtime
[junit4:junit4] 2> 743629 T1868 C473 P33034 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 743630 T1868 C473 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 743630 T1892 C472 P46648 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 743630 T1892 C472 P46648 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 743632 T1867 C473 P33034 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 743632 T1867 C473 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 743633 T1892 C472 P46648 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 743633 T1892 C472 P46648 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 743634 T1892 C472 P46648 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 743635 T1869 C473 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 743636 T1892 C472 P46648 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 743637 T1892 C472 P46648 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index.20130623201143184
[junit4:junit4] 2> 743637 T1892 C472 P46648 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index.20130623201143184 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bd1264; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 743639 T1868 C473 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 743640 T1892 C472 P46648 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 743648 T1892 C472 P46648 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 743648 T1892 C472 P46648 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 743650 T1892 C472 P46648 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 743650 T1892 C472 P46648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 743651 T1892 C472 P46648 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 743651 T1892 C472 P46648 oass.SolrIndexSearcher.<init> Opening Searcher@165c08a main
[junit4:junit4] 2> 743652 T1891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@165c08a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 743652 T1892 C472 P46648 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index.20130623201143184 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index.20130623201143184;done=true>>]
[junit4:junit4] 2> 743653 T1892 C472 P46648 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index.20130623201143184
[junit4:junit4] 2> 743653 T1892 C472 P46648 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index.20130623201143184
[junit4:junit4] 2> 743653 T1892 C472 P46648 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 743653 T1892 C472 P46648 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 743654 T1892 C472 P46648 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 743654 T1892 C472 P46648 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 743668 T1892 C472 P46648 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 744129 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 744130 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48600_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48600"}
[junit4:junit4] 2> 744130 T1854 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 744130 T1854 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 744133 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:46648__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46648_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46648"}
[junit4:junit4] 2> 744136 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 744136 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 744136 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 744137 T1906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 744137 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 744136 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 744617 T1925 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/collection1
[junit4:junit4] 2> 744618 T1925 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 744618 T1925 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 744618 T1925 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 744619 T1925 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/collection1/'
[junit4:junit4] 2> 744620 T1925 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/collection1/lib/README' to classloader
[junit4:junit4] 2> 744620 T1925 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 744654 T1925 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 744694 T1925 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 744695 T1925 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 744700 T1925 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 745132 T1925 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 745132 T1925 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 745133 T1925 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 745140 T1925 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 745143 T1925 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 745151 T1925 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 745154 T1925 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 745159 T1925 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 745159 T1925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 745160 T1925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 745160 T1925 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 745161 T1925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 745161 T1925 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 745162 T1925 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 745162 T1925 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372007502005/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/
[junit4:junit4] 2> 745162 T1925 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497062
[junit4:junit4] 2> 745163 T1925 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 745163 T1925 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4
[junit4:junit4] 2> 745163 T1925 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index/
[junit4:junit4] 2> 745164 T1925 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 745164 T1925 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index
[junit4:junit4] 2> 745167 T1925 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 745167 T1925 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 745169 T1925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 745169 T1925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 745170 T1925 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 745170 T1925 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 745171 T1925 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 745171 T1925 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 745172 T1925 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 745172 T1925 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 745173 T1925 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 745175 T1925 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 745178 T1925 oass.SolrIndexSearcher.<init> Opening Searcher@1c7e73f main
[junit4:junit4] 2> 745178 T1925 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 745179 T1925 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 745184 T1927 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c7e73f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 745186 T1925 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 745186 T1925 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48600 collection:collection1 shard:shard1
[junit4:junit4] 2> 745189 T1925 oasc.ZkController.register We are http://127.0.0.1:48600/collection1/ and leader is http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 745189 T1925 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48600
[junit4:junit4] 2> 745189 T1925 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 745189 T1925 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C474 name=collection1 org.apache.solr.core.SolrCore@1671af2 url=http://127.0.0.1:48600/collection1 node=127.0.0.1:48600_ C474_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:48600_, base_url=http://127.0.0.1:48600}
[junit4:junit4] 2> 745190 T1928 C474 P48600 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 745190 T1925 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 745190 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 745191 T1928 C474 P48600 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 745191 T1928 C474 P48600 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 745191 T1831 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 745192 T1831 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 745192 T1928 C474 P48600 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 745192 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 745197 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 745197 T1867 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:48600__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 745198 T1831 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 745198 T1831 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 745199 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C475 name=collection1 org.apache.solr.core.SolrCore@18bb8a url=http://127.0.0.1:38345/collection1 node=127.0.0.1:38345_ C475_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:38345_, base_url=http://127.0.0.1:38345}
[junit4:junit4] 2> 745459 T1909 C475 P38345 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33034/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 745460 T1909 C475 P38345 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38345 START replicas=[http://127.0.0.1:33034/collection1/] nUpdates=100
[junit4:junit4] 2> 745460 T1909 C475 P38345 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 745460 T1909 C475 P38345 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 745460 T1909 C475 P38345 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 745461 T1909 C475 P38345 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 745461 T1909 C475 P38345 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 745461 T1909 C475 P38345 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33034/collection1/. core=collection1
[junit4:junit4] 2> 745461 T1909 C475 P38345 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C476 name=collection1 org.apache.solr.core.SolrCore@537fb2 url=http://127.0.0.1:33034/collection1 node=127.0.0.1:33034_ C476_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:33034_, base_url=http://127.0.0.1:33034, leader=true}
[junit4:junit4] 2> 745462 T1868 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 745465 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 745478 T1866 C476 P33034 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 745478 T1866 C476 P33034 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 745479 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 745479 T1866 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4] 2> 745480 T1909 C475 P38345 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 745480 T1909 C475 P38345 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 745482 T1868 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 745482 T1909 C475 P38345 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 745482 T1909 C475 P38345 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 745483 T1909 C475 P38345 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 745484 T1866 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 745485 T1909 C475 P38345 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 745485 T1909 C475 P38345 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index.20130623201145033
[junit4:junit4] 2> 745486 T1909 C475 P38345 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index.20130623201145033 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2b81de; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 745488 T1869 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 745489 T1909 C475 P38345 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 745491 T1909 C475 P38345 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 745491 T1909 C475 P38345 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 745493 T1909 C475 P38345 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 745493 T1909 C475 P38345 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 745493 T1909 C475 P38345 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 745494 T1909 C475 P38345 oass.SolrIndexSearcher.<init> Opening Searcher@15cebe6 main
[junit4:junit4] 2> 745495 T1908 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15cebe6 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 745495 T1909 C475 P38345 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index.20130623201145033 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index.20130623201145033;done=true>>]
[junit4:junit4] 2> 745495 T1909 C475 P38345 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index.20130623201145033
[junit4:junit4] 2> 745496 T1909 C475 P38345 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index.20130623201145033
[junit4:junit4] 2> 745496 T1909 C475 P38345 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 745496 T1909 C475 P38345 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 745496 T1909 C475 P38345 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 745497 T1909 C475 P38345 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 745518 T1909 C475 P38345 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 745646 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 745647 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48600__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48600_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48600"}
[junit4:junit4] 2> 745650 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38345__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38345_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38345"}
[junit4:junit4] 2> 745652 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 745652 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 745653 T1906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 745652 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 745652 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 745652 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 746198 T1867 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:48600__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 746198 T1867 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:48600__collection1&state=recovering&nodeName=127.0.0.1:48600_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 746200 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 747201 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C474_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:48600_, base_url=http://127.0.0.1:48600}
[junit4:junit4] 2> 748199 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33034/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 748199 T1928 C474 P48600 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48600 START replicas=[http://127.0.0.1:33034/collection1/] nUpdates=100
[junit4:junit4] 2> 748200 T1928 C474 P48600 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 748200 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 748200 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 748200 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 748200 T1928 C474 P48600 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 748201 T1928 C474 P48600 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33034/collection1/. core=collection1
[junit4:junit4] 2> 748201 T1928 C474 P48600 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 748201 T1867 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 748203 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 748204 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 748208 T1866 C476 P33034 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 748208 T1866 C476 P33034 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 748209 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 748209 T1866 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 748209 T1928 C474 P48600 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 748210 T1928 C474 P48600 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 748211 T1867 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 748212 T1928 C474 P48600 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 748212 T1928 C474 P48600 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 748212 T1928 C474 P48600 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 748214 T1868 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0
[junit4:junit4] 2> 748214 T1928 C474 P48600 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 748215 T1928 C474 P48600 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index.20130623201147763
[junit4:junit4] 2> 748215 T1928 C474 P48600 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index.20130623201147763 lockFactory=org.apache.lucene.store.NativeFSLockFactory@143e7f9; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 748218 T1866 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 748219 T1928 C474 P48600 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 748221 T1928 C474 P48600 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 748221 T1928 C474 P48600 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 748222 T1928 C474 P48600 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 748223 T1928 C474 P48600 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 748223 T1928 C474 P48600 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 748223 T1928 C474 P48600 oass.SolrIndexSearcher.<init> Opening Searcher@1868aa7 main
[junit4:junit4] 2> 748224 T1927 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1868aa7 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 748225 T1928 C474 P48600 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index.20130623201147763 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index.20130623201147763;done=true>>]
[junit4:junit4] 2> 748225 T1928 C474 P48600 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index.20130623201147763
[junit4:junit4] 2> 748225 T1928 C474 P48600 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index.20130623201147763
[junit4:junit4] 2> 748226 T1928 C474 P48600 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 748226 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 748226 T1928 C474 P48600 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 748226 T1928 C474 P48600 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 748227 T1928 C474 P48600 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 748660 T1854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 748660 T1854 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48600__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48600_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48600"}
[junit4:junit4] 2> 748681 T1889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 748682 T1906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 748681 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 748681 T1923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 748681 T1860 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 748681 T1853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 749204 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 749205 T1831 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C477 name=collection1 org.apache.solr.core.SolrCore@1b6bc9e url=http://127.0.0.1:49240/collection1 node=127.0.0.1:49240_ C477_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:49240_, base_url=http://127.0.0.1:49240, leader=true}
[junit4:junit4] 2> 749211 T1847 C477 P49240 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 749213 T1847 C477 P49240 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6899a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 749214 T1847 C477 P49240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 749218 T1847 C477 P49240 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6899a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6899a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 749218 T1847 C477 P49240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 749219 T1847 C477 P49240 oass.SolrIndexSearcher.<init> Opening Searcher@1597dc9 main
[junit4:junit4] 2> 749219 T1847 C477 P49240 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 749221 T1857 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1597dc9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 749222 T1847 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> C474_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:48600_, base_url=http://127.0.0.1:48600}
[junit4:junit4] 2> 749229 T1918 C474 P48600 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:33034/collection1/, StdNode: http://127.0.0.1:46648/collection1/, StdNode: http://127.0.0.1:38345/collection1/, StdNode: http://127.0.0.1:48600/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 749232 T1919 C474 P48600 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 749232 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C478 name=collection1 org.apache.solr.core.SolrCore@89f479 url=http://127.0.0.1:46648/collection1 node=127.0.0.1:46648_ C478_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:46648_, base_url=http://127.0.0.1:46648}
[junit4:junit4] 2> 749237 T1885 C478 P46648 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C475_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38345_, base_url=http://127.0.0.1:38345}
[junit4:junit4] 2> 749240 T1902 C475 P38345 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 749258 T1866 C476 P33034 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 749258 T1866 C476 P33034 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 749259 T1919 C474 P48600 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 749259 T1866 C476 P33034 oass.SolrIndexSearcher.<init> Opening Searcher@731200 main
[junit4:junit4] 2> 749259 T1919 C474 P48600 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 749260 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 749260 T1919 C474 P48600 oass.SolrIndexSearcher.<init> Opening Searcher@3fec19 main
[junit4:junit4] 2> 749261 T1919 C474 P48600 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 749261 T1875 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@731200 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 749262 T1927 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fec19 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 749262 T1866 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4] 2> 749262 T1919 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4] 2> 749278 T1885 C478 P46648 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 749278 T1885 C478 P46648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 749278 T1902 C475 P38345 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 749279 T1885 C478 P46648 oass.SolrIndexSearcher.<init> Opening Searcher@f77652 main
[junit4:junit4] 2> 749279 T1902 C475 P38345 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 749280 T1885 C478 P46648 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 749281 T1891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f77652 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 749281 T1902 C475 P38345 oass.SolrIndexSearcher.<init> Opening Searcher@199af35 main
[junit4:junit4] 2> 749281 T1885 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4] 2> 749281 T1902 C475 P38345 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 749283 T1908 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@199af35 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 749283 T1902 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 43
[junit4:junit4] 2> 749284 T1918 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 55
[junit4:junit4] 2> 749285 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 749286 T1867 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 749288 T1884 C478 P46648 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 749289 T1899 C475 P38345 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 749290 T1916 C474 P48600 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 751294 T1846 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438654147686563840)} 0 2
[junit4:junit4] 2> 751297 T1899 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438654147689709568&update.from=http://127.0.0.1:33034/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438654147689709568)} 0 0
[junit4:junit4] 2> 751298 T1916 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438654147689709568&update.from=http://127.0.0.1:33034/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438654147689709568)} 0 1
[junit4:junit4] 2> 751298 T1885 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438654147689709568&update.from=http://127.0.0.1:33034/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438654147689709568)} 0 1
[junit4:junit4] 2> 751299 T1867 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438654147689709568)} 0 4
[junit4:junit4] 2> 751302 T1847 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438654147696001024)]} 0 2
[junit4:junit4] 2> 751309 T1900 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438654147700195328)]} 0 1
[junit4:junit4] 2> 751309 T1884 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438654147700195328)]} 0 1
[junit4:junit4] 2> 751309 T1918 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438654147700195328)]} 0 2
[junit4:junit4] 2> 751310 T1868 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:48600/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438654147700195328)]} 0 5
[junit4:junit4] 2> 751311 T1919 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 7
[junit4:junit4] 2> 751313 T1848 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438654147708583936)]} 0 0
[junit4:junit4] 2> 751318 T1917 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438654147711729664)]} 0 0
[junit4:junit4] 2> 751319 T1902 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438654147711729664)]} 0 1
[junit4:junit4] 2> 751318 T1882 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438654147711729664)]} 0 0
[junit4:junit4] 2> 751321 T1869 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:38345/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438654147711729664)]} 0 5
[junit4:junit4] 2> 751322 T1901 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 8
[junit4:junit4] 2> 751324 T1849 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438654147720118272)]} 0 0
[junit4:junit4] 2> 751327 T1867 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:46648/collection1/]
[junit4:junit4] 2> 751327 T1867 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:46648/collection1/ against:[http://127.0.0.1:46648/collection1/] result:true
[junit4:junit4] 2> 751328 T1867 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:38345/collection1/ against:[http://127.0.0.1:46648/collection1/] result:false
[junit4:junit4] 2> 751328 T1867 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:48600/collection1/ against:[http://127.0.0.1:46648/collection1/] result:false
[junit4:junit4] 2> 751331 T1916 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&wt=javabin&version=2} {add=[2 (1438654147724312576)]} 0 1
[junit4:junit4] 2> 751331 T1899 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&wt=javabin&version=2} {add=[2 (1438654147724312576)]} 0 1
[junit4:junit4] 2> 751332 T1867 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&wt=javabin&version=2} {add=[2 (1438654147724312576)]} 0 6
[junit4:junit4] 2> 751334 T1846 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438654147730604032)]} 0 0
[junit4:junit4] 2> 751338 T1868 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:46648/collection1/]
[junit4:junit4] 2> 751338 T1868 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:46648/collection1/ against:[http://127.0.0.1:46648/collection1/] result:true
[junit4:junit4] 2> 751338 T1868 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:38345/collection1/ against:[http://127.0.0.1:46648/collection1/] result:false
[junit4:junit4] 2> 751339 T1868 C476 P33034 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:48600/collection1/ against:[http://127.0.0.1:46648/collection1/] result:false
[junit4:junit4] 2> 751341 T1900 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&wt=javabin&version=2} {add=[3 (1438654147735846912)]} 0 0
[junit4:junit4] 2> 751342 T1919 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:33034/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&wt=javabin&version=2} {add=[3 (1438654147735846912)]} 0 1
[junit4:junit4] 2> 751342 T1868 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:48600/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&wt=javabin&version=2} {add=[3 (1438654147735846912)]} 0 4
[junit4:junit4] 2> 751343 T1918 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:46648/collection1/&test.distrib.skip.servers=http://127.0.0.1:38345/collection1/&wt=javabin&version=2} {add=[3]} 0 7
[junit4:junit4] 2> 751345 T1847 C477 P49240 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 751358 T1847 C477 P49240 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6899a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6899a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 751358 T1847 C477 P49240 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 751360 T1847 C477 P49240 oass.SolrIndexSearcher.<init> Opening Searcher@624824 main
[junit4:junit4] 2> 751360 T1847 C477 P49240 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 751361 T1857 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@624824 main{StandardDirectoryReader(segments_3:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 751361 T1847 C477 P49240 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 751363 T1883 C478 P46648 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:33034/collection1/, StdNode: http://127.0.0.1:46648/collection1/, StdNode: http://127.0.0.1:38345/collection1/, StdNode: http://127.0.0.1:48600/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 751366 T1885 C478 P46648 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 751366 T1917 C474 P48600 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 751366 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 751367 T1901 C475 P38345 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 751418 T1917 C474 P48600 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@120354; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 751418 T1917 C474 P48600 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 751418 T1866 C476 P33034 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@f99bfd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 751419 T1866 C476 P33034 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 751418 T1885 C478 P46648 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a14cc6; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 751420 T1885 C478 P46648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 751420 T1917 C474 P48600 oass.SolrIndexSearcher.<init> Opening Searcher@1abc307 main
[junit4:junit4] 2> 751421 T1866 C476 P33034 oass.SolrIndexSearcher.<init> Opening Searcher@15d1003 main
[junit4:junit4] 2> 751423 T1927 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1abc307 main{StandardDirectoryReader(segments_6:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 751422 T1885 C478 P46648 oass.SolrIndexSearcher.<init> Opening Searcher@20745c main
[junit4:junit4] 2> 751421 T1917 C474 P48600 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 751424 T1885 C478 P46648 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 751423 T1866 C476 P33034 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 751425 T1875 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15d1003 main{StandardDirectoryReader(segments_6:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 751424 T1917 C474 P48600 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 58
[junit4:junit4] 2> 751425 T1891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20745c main{StandardDirectoryReader(segments_4:3:nrt _0(4.4):c2)}
[junit4:junit4] 2> 751426 T1866 C476 P33034 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 60
[junit4:junit4] 2> 751427 T1885 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 61
[junit4:junit4] 2> 751428 T1901 C475 P38345 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1372007488196/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1af34ba; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 751428 T1901 C475 P38345 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 751429 T1901 C475 P38345 oass.SolrIndexSearcher.<init> Opening Searcher@1230a0f main
[junit4:junit4] 2> 751430 T1901 C475 P38345 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 751431 T1908 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1230a0f main{StandardDirectoryReader(segments_5:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 751431 T1901 C475 P38345 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 65
[junit4:junit4] 2> 751432 T1883 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 69
[junit4:junit4] 2> 751432 T1831 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 751433 T1831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 751434 T1831 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 751436 T1869 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 751438 T1884 C478 P46648 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 751440 T1899 C475 P38345 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 751442 T1916 C474 P48600 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 751444 T1831 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 751449 T1900 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 751449 T1900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 751453 T1867 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 751453 T1867 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33034/collection1/
[junit4:junit4] 2> 751453 T1867 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33034 START replicas=[http://127.0.0.1:46648/collection1/, http://127.0.0.1:38345/collection1/, http://127.0.0.1:48600/collection1/] nUpdates=100
[junit4:junit4] 2> 751455 T1882 C478 P46648 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751455 T1902 C475 P38345 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751455 T1919 C474 P48600 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751455 T1867 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33034 Received 3 versions from 127.0.0.1:46648/collection1/
[junit4:junit4] 2> 751456 T1867 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33034 Our versions are newer. ourLowThreshold=1438654147689709568 otherHigh=1438654147711729664
[junit4:junit4] 2> 751456 T1867 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33034 Received 5 versions from 127.0.0.1:38345/collection1/
[junit4:junit4] 2> 751457 T1867 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33034 Our versions are newer. ourLowThreshold=1438654147689709568 otherHigh=1438654147724312576
[junit4:junit4] 2> 751457 T1867 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33034 Received 5 versions from 127.0.0.1:48600/collection1/
[junit4:junit4] 2> 751457 T1867 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:33034 Our versions are newer. ourLowThreshold=1438654147689709568 otherHigh=1438654147724312576
[junit4:junit4] 2> 751458 T1867 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33034 DONE. sync succeeded
[junit4:junit4] 2> 751458 T1867 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 751458 T1867 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/: try and ask http://127.0.0.1:46648/collection1/ to sync
[junit4:junit4] 2> 751459 T1867 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/: try and ask http://127.0.0.1:38345/collection1/ to sync
[junit4:junit4] 2> 751459 T1867 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/: try and ask http://127.0.0.1:48600/collection1/ to sync
[junit4:junit4] 2> 751460 T1885 C478 P46648 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46648 START replicas=[http://127.0.0.1:33034/collection1/] nUpdates=100
[junit4:junit4] 2> 751460 T1901 C475 P38345 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38345 START replicas=[http://127.0.0.1:33034/collection1/] nUpdates=100
[junit4:junit4] 2> 751460 T1918 C474 P48600 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48600 START replicas=[http://127.0.0.1:33034/collection1/] nUpdates=100
[junit4:junit4] 2> 751461 T1868 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751462 T1866 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751462 T1901 C475 P38345 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38345 Received 5 versions from 127.0.0.1:33034/collection1/
[junit4:junit4] 2> 751462 T1869 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751462 T1901 C475 P38345 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38345 Our versions are newer. ourLowThreshold=1438654147689709568 otherHigh=1438654147724312576
[junit4:junit4] 2> 751463 T1901 C475 P38345 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38345 DONE. sync succeeded
[junit4:junit4] 2> 751462 T1918 C474 P48600 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:48600 Received 5 versions from 127.0.0.1:33034/collection1/
[junit4:junit4] 2> 751464 T1901 C475 P38345 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:33034/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4
[junit4:junit4] 2> 751463 T1885 C478 P46648 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46648 Received 5 versions from 127.0.0.1:33034/collection1/
[junit4:junit4] 2> 751464 T1867 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/: sync completed with http://127.0.0.1:38345/collection1/
[junit4:junit4] 2> 751464 T1918 C474 P48600 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:48600 Our versions are newer. ourLowThreshold=1438654147689709568 otherHigh=1438654147724312576
[junit4:junit4] 2> 751465 T1918 C474 P48600 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48600 DONE. sync succeeded
[junit4:junit4] 2> 751465 T1885 C478 P46648 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:46648 Requesting updates from 127.0.0.1:33034/collection1/n=2 versions=[1438654147735846912, 1438654147724312576]
[junit4:junit4] 2> 751466 T1918 C474 P48600 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:33034/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 751466 T1867 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/: sync completed with http://127.0.0.1:48600/collection1/
[junit4:junit4] 2> 751468 T1868 C476 P33034 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1438654147735846912,1438654147724312576&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 751470 T1885 C478 P46648 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438654147724312576), 3 (1438654147735846912)]} 0 2
[junit4:junit4] 2> 751471 T1885 C478 P46648 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46648 DONE. sync succeeded
[junit4:junit4] 2> 751471 T1885 C478 P46648 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:33034/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=12
[junit4:junit4] 2> 751471 T1867 oasc.SyncStrategy.syncToMe http://127.0.0.1:33034/collection1/: sync completed with http://127.0.0.1:46648/collection1/
[junit4:junit4] 2> 751472 T1867 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=19
[junit4:junit4] 2> 751472 T1900 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&acti
[...truncated too long message...]
y@173d25a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9170d8),segFN=segments_3,generation=3}
[junit4:junit4] 2> 928485 T2806 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 928486 T2806 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 928486 T2806 oass.SolrIndexSearcher.<init> Opening Searcher@1a52aae main
[junit4:junit4] 2> 928486 T2805 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a52aae main{StandardDirectoryReader(segments_3:4:nrt _0(4.4):C2)}
[junit4:junit4] 2> 928487 T2806 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index.20130624041448005 [CachedDir<<refCount=0;path=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index.20130624041448005;done=true>>]
[junit4:junit4] 2> 928487 T2806 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index.20130624041448005
[junit4:junit4] 2> 928487 T2806 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index.20130624041448005
[junit4:junit4] 2> 928573 T2800 C566 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={sort=id+desc&q=*:*&wt=javabin&version=2} hits=2 status=0 QTime=0
[junit4:junit4] 2> 928574 T2178 oas.SolrTestCaseJ4.tearDown ###Ending doTestReplicateAfterCoreReload
[junit4:junit4] 2> 928594 T2178 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=3149183
[junit4:junit4] 2> 928595 T2178 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@14a6fd1
[junit4:junit4] 2> 928596 T2178 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 928596 T2178 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 928597 T2178 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 928597 T2178 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 928598 T2178 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 928599 T2178 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 928599 T2178 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686214/master/collection1/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686214/master/collection1/data;done=false>>]
[junit4:junit4] 2> 928599 T2178 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686214/master/collection1/data
[junit4:junit4] 2> 928600 T2178 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686214/master/collection1/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686214/master/collection1/data/index;done=false>>]
[junit4:junit4] 2> 928600 T2178 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686214/master/collection1/data/index
[junit4:junit4] 2> 928601 T2178 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 928673 T2178 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=21161737
[junit4:junit4] 2> 928674 T2178 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1948a9
[junit4:junit4] 2> 928674 T2178 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 928675 T2178 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 928675 T2178 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 928676 T2178 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 928677 T2178 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 928677 T2178 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 928677 T2178 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index;done=false>>]
[junit4:junit4] 2> 928678 T2178 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data/index
[junit4:junit4] 2> 928678 T2178 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data;done=false>>]
[junit4:junit4] 2> 928679 T2178 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1372007686381/slave/collection1/data
[junit4:junit4] 2> 928679 T2178 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 928733 T2178 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {newname=PostingsFormat(name=Direct), id=PostingsFormat(name=Direct), name=PostingsFormat(name=Direct)}, docValues:{}, sim=DefaultSimilarity, locale=fi, timezone=Pacific/Ponape
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=1,free=26884192,total=257884160
[junit4:junit4] 2> NOTE: All tests run in this JVM: [DirectUpdateHandlerTest, TestSolrXMLSerializer, CurrencyFieldXmlFileTest, HardAutoCommitTest, HighlighterConfigTest, TestDFRSimilarityFactory, TestHashPartitioner, StandardRequestHandlerTest, TestDistributedSearch, TestWordDelimiterFilterFactory, TestShardHandlerFactory, TestSolrQueryParserResource, SortByFunctionTest, TestLMJelinekMercerSimilarityFactory, TestRTGBase, SimplePostToolTest, TestJmxIntegration, FieldAnalysisRequestHandlerTest, LeaderElectionIntegrationTest, TestSystemIdResolver, SpellCheckComponentTest, OpenExchangeRatesOrgProviderTest, TestBM25SimilarityFactory, XmlUpdateRequestHandlerTest, ZkNodePropsTest, TestZkChroot, TestArbitraryIndexDir, TestMergePolicyConfig, NotRequiredUniqueKeyTest, TestStressReorder, CopyFieldTest, SampleTest, TestComponentsName, TestPropInject, OpenCloseCoreStressTest, CurrencyFieldOpenExchangeTest, BasicZkTest, TestLazyCores, JSONWriterTest, ClusterStateTest, TestCopyFieldCollectionResource, SignatureUpdateProcessorFactoryTest, DefaultValueUpdateProcessorTest, TestSuggestSpellingConverter, TestCSVResponseWriter, XsltUpdateRequestHandlerTest, SuggesterTSTTest, IndexBasedSpellCheckerTest, TestPartialUpdateDeduplication, DOMUtilTest, ConvertedLegacyTest, TestStressLucene, TestSolrJ, TestFieldTypeResource, PreAnalyzedUpdateProcessorTest, UniqFieldsUpdateProcessorFactoryTest, TestUtils, PathHierarchyTokenizerFactoryTest, TestBinaryResponseWriter, TestSchemaNameResource, TestFiltering, TestCoreDiscovery, TestUpdate, HighlighterTest, DocValuesMultiTest, UpdateParamsTest, TestAnalyzedSuggestions, TestCodecSupport, SolrCmdDistributorTest, TestFastOutputStream, BadIndexSchemaTest, TestManagedSchemaFieldResource, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, SyncSliceTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler]
[junit4:junit4] Completed on J1 in 64.88s, 13 tests, 1 failure <<< FAILURES!
[...truncated 445 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1256 tests, 3 failures, 13 ignored (7 assumptions)
Total time: 42 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure