You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/19 00:33:55 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b93) - Build #
6127 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6127/
Java: 64bit/jdk1.8.0-ea-b93 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 30 from http://127.0.0.1:34724/c_qo/w/collection1lastClient and got 29 from http://127.0.0.1:50639/c_qo/w/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 30 from http://127.0.0.1:34724/c_qo/w/collection1lastClient and got 29 from http://127.0.0.1:50639/c_qo/w/collection1
at __randomizedtesting.SeedInfo.seed([F73BF3A867A3596E:76DD7DB010FC3952]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9730 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 648510 T1229 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /c_qo/w
[junit4:junit4] 2> 648514 T1229 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371594417648
[junit4:junit4] 2> 648515 T1229 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 648515 T1230 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 648615 T1229 oasc.ZkTestServer.run start zk server on port:56626
[junit4:junit4] 2> 648617 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 648677 T1236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@223c4bee name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 648677 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 648677 T1229 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 648682 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 648683 T1238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cfbb1d9 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 648683 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 648683 T1229 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 648686 T1229 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 648688 T1229 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 648690 T1229 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 648692 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 648692 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 648696 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 648697 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 648700 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 648700 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 648703 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 648703 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 648705 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 648706 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 648708 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 648709 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 648712 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 648712 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 648716 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 648716 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 648719 T1229 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 648719 T1229 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 648800 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 648819 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53082
[junit4:junit4] 2> 648820 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 648820 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 648820 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856
[junit4:junit4] 2> 648829 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/solr.xml
[junit4:junit4] 2> 648829 T1229 oasc.CoreContainer.<init> New CoreContainer 1428745281
[junit4:junit4] 2> 648829 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/'
[junit4:junit4] 2> 648830 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/'
[junit4:junit4] 2> 648868 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 648869 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 648869 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 648869 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 648869 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 648870 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 648870 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 648870 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 648870 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 648870 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 648873 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 648874 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 648874 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 648875 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 648877 T1249 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ed066f2 name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 648877 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 648879 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 648882 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 648884 T1251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7691fc1 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 648884 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 648887 T1229 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 648890 T1229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 648893 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 648895 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53082_c_qo%2Fw
[junit4:junit4] 2> 648895 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53082_c_qo%2Fw
[junit4:junit4] 2> 648898 T1229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 648903 T1229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 648905 T1229 oasc.Overseer.start Overseer (id=89888811755110403-127.0.0.1:53082_c_qo%2Fw-n_0000000000) starting
[junit4:junit4] 2> 648908 T1229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 648912 T1253 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 648912 T1229 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 648915 T1229 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 648917 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 648920 T1252 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 648922 T1254 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 648922 T1254 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 650425 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 650426 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53082/c_qo/w",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53082_c_qo%2Fw"}
[junit4:junit4] 2> 650427 T1252 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 650427 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 650434 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 650923 T1254 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/collection1
[junit4:junit4] 2> 650924 T1254 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 650925 T1254 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 650925 T1254 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 650927 T1254 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/collection1/'
[junit4:junit4] 2> 650928 T1254 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/collection1/lib/README' to classloader
[junit4:junit4] 2> 650928 T1254 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 650973 T1254 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 651006 T1254 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 651007 T1254 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 651011 T1254 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 651352 T1254 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 651356 T1254 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 651358 T1254 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 651372 T1254 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 651375 T1254 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 651379 T1254 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 651380 T1254 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 651380 T1254 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 651380 T1254 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 651381 T1254 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 651381 T1254 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 651381 T1254 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 651389 T1254 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371594417856/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/control/data/
[junit4:junit4] 2> 651390 T1254 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 651390 T1254 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 651391 T1254 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/control/data
[junit4:junit4] 2> 651391 T1254 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/control/data/index/
[junit4:junit4] 2> 651391 T1254 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 651392 T1254 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/control/data/index
[junit4:junit4] 2> 651395 T1254 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 651395 T1254 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 651397 T1254 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 651398 T1254 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 651398 T1254 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 651399 T1254 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 651399 T1254 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 651399 T1254 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 651400 T1254 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 651400 T1254 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 651400 T1254 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 651403 T1254 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 651407 T1254 oass.SolrIndexSearcher.<init> Opening Searcher@4c45f048 main
[junit4:junit4] 2> 651407 T1254 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 651408 T1254 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 651411 T1255 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c45f048 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 651412 T1254 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 651412 T1254 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53082/c_qo/w collection:control_collection shard:shard1
[junit4:junit4] 2> 651412 T1254 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 651418 T1254 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 651421 T1254 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 651421 T1254 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 651421 T1254 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53082/c_qo/w/collection1/
[junit4:junit4] 2> 651422 T1254 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 651422 T1254 oasc.SyncStrategy.syncToMe http://127.0.0.1:53082/c_qo/w/collection1/ has no replicas
[junit4:junit4] 2> 651422 T1254 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53082/c_qo/w/collection1/
[junit4:junit4] 2> 651422 T1254 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 651940 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 651953 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 651981 T1254 oasc.ZkController.register We are http://127.0.0.1:53082/c_qo/w/collection1/ and leader is http://127.0.0.1:53082/c_qo/w/collection1/
[junit4:junit4] 2> 651981 T1254 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53082/c_qo/w
[junit4:junit4] 2> 651982 T1254 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 651982 T1254 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 651982 T1254 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 651985 T1254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 651986 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 651987 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 651988 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 651994 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 651996 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 651999 T1258 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7aa780f3 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 652000 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 652002 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 652007 T1229 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 652077 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 652078 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46219
[junit4:junit4] 2> 652079 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 652079 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 652079 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141
[junit4:junit4] 2> 652080 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/solr.xml
[junit4:junit4] 2> 652080 T1229 oasc.CoreContainer.<init> New CoreContainer 156559106
[junit4:junit4] 2> 652080 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/'
[junit4:junit4] 2> 652080 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/'
[junit4:junit4] 2> 652119 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 652120 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 652120 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 652120 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 652120 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 652121 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 652121 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 652121 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 652121 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 652122 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 652125 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 652125 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 652125 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 652126 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 652128 T1269 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d4c544f name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 652129 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 652130 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 652134 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 652135 T1271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6aca274f name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 652135 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 652139 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 653142 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46219_c_qo%2Fw
[junit4:junit4] 2> 653144 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46219_c_qo%2Fw
[junit4:junit4] 2> 653149 T1271 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 653149 T1258 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 653149 T1251 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 653150 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 653156 T1272 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 653156 T1272 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 653476 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 653477 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53082/c_qo/w",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53082_c_qo%2Fw"}
[junit4:junit4] 2> 653484 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46219/c_qo/w",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:46219_c_qo%2Fw"}
[junit4:junit4] 2> 653484 T1252 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 653484 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 653493 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 653493 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 653493 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 654157 T1272 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/collection1
[junit4:junit4] 2> 654158 T1272 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 654159 T1272 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 654159 T1272 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 654161 T1272 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/collection1/'
[junit4:junit4] 2> 654162 T1272 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/collection1/lib/README' to classloader
[junit4:junit4] 2> 654163 T1272 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 654212 T1272 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 654241 T1272 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 654243 T1272 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 654246 T1272 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 654564 T1272 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 654569 T1272 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 654571 T1272 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 654583 T1272 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 654587 T1272 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 654589 T1272 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 654590 T1272 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 654591 T1272 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 654591 T1272 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 654592 T1272 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 654592 T1272 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 654592 T1272 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 654592 T1272 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371594421141/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/
[junit4:junit4] 2> 654593 T1272 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 654593 T1272 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 654593 T1272 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1
[junit4:junit4] 2> 654594 T1272 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index/
[junit4:junit4] 2> 654594 T1272 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 654594 T1272 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index
[junit4:junit4] 2> 654597 T1272 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 654597 T1272 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 654599 T1272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 654599 T1272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 654600 T1272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 654601 T1272 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 654601 T1272 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 654601 T1272 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 654601 T1272 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 654602 T1272 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 654602 T1272 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 654603 T1272 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 654606 T1272 oass.SolrIndexSearcher.<init> Opening Searcher@6673b68f main
[junit4:junit4] 2> 654607 T1272 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 654607 T1272 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 654611 T1273 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6673b68f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 654613 T1272 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 654613 T1272 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46219/c_qo/w collection:collection1 shard:shard2
[junit4:junit4] 2> 654614 T1272 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 654628 T1272 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 654630 T1272 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 654630 T1272 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 654630 T1272 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46219/c_qo/w/collection1/
[junit4:junit4] 2> 654630 T1272 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 654631 T1272 oasc.SyncStrategy.syncToMe http://127.0.0.1:46219/c_qo/w/collection1/ has no replicas
[junit4:junit4] 2> 654631 T1272 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46219/c_qo/w/collection1/
[junit4:junit4] 2> 654631 T1272 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 655002 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 655015 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 655015 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 655015 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 655036 T1272 oasc.ZkController.register We are http://127.0.0.1:46219/c_qo/w/collection1/ and leader is http://127.0.0.1:46219/c_qo/w/collection1/
[junit4:junit4] 2> 655037 T1272 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46219/c_qo/w
[junit4:junit4] 2> 655037 T1272 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 655037 T1272 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 655037 T1272 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 655038 T1272 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 655039 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 655040 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 655040 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 655132 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 655134 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40299
[junit4:junit4] 2> 655135 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 655135 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 655136 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176
[junit4:junit4] 2> 655136 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/solr.xml
[junit4:junit4] 2> 655136 T1229 oasc.CoreContainer.<init> New CoreContainer 1187059526
[junit4:junit4] 2> 655137 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/'
[junit4:junit4] 2> 655137 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/'
[junit4:junit4] 2> 655176 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 655176 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 655176 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 655177 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 655177 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 655177 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 655177 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 655178 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 655178 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 655178 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 655181 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 655181 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 655182 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 655182 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 655184 T1285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f7623cc name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 655184 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 655186 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 655189 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 655190 T1287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46313d02 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 655190 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 655194 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 656198 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40299_c_qo%2Fw
[junit4:junit4] 2> 656200 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40299_c_qo%2Fw
[junit4:junit4] 2> 656205 T1271 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 656205 T1251 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 656205 T1258 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 656205 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 656205 T1287 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 656206 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 656205 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 656216 T1288 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 656216 T1288 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 656526 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 656527 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46219/c_qo/w",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:46219_c_qo%2Fw"}
[junit4:junit4] 2> 656531 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40299/c_qo/w",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:40299_c_qo%2Fw"}
[junit4:junit4] 2> 656531 T1252 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 656531 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 656536 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 656536 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 656536 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 656536 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 657219 T1288 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/collection1
[junit4:junit4] 2> 657219 T1288 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 657221 T1288 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 657221 T1288 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 657223 T1288 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/collection1/'
[junit4:junit4] 2> 657224 T1288 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/collection1/lib/README' to classloader
[junit4:junit4] 2> 657224 T1288 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 657288 T1288 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 657317 T1288 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 657318 T1288 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 657322 T1288 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 657673 T1288 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 657679 T1288 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 657682 T1288 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 657699 T1288 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 657706 T1288 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 657711 T1288 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 657716 T1288 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 657717 T1288 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 657717 T1288 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 657719 T1288 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 657719 T1288 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 657719 T1288 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 657719 T1288 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371594424176/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/
[junit4:junit4] 2> 657719 T1288 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 657720 T1288 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 657721 T1288 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2
[junit4:junit4] 2> 657721 T1288 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index/
[junit4:junit4] 2> 657721 T1288 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 657724 T1288 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index
[junit4:junit4] 2> 657728 T1288 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 657729 T1288 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 657731 T1288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 657731 T1288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 657731 T1288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 657732 T1288 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 657733 T1288 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 657733 T1288 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 657733 T1288 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 657734 T1288 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 657734 T1288 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 657736 T1288 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 657739 T1288 oass.SolrIndexSearcher.<init> Opening Searcher@a83ca2e main
[junit4:junit4] 2> 657740 T1288 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 657740 T1288 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 657753 T1289 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a83ca2e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 657755 T1288 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 657755 T1288 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40299/c_qo/w collection:collection1 shard:shard3
[junit4:junit4] 2> 657756 T1288 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 657761 T1288 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 657763 T1288 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 657763 T1288 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 657763 T1288 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40299/c_qo/w/collection1/
[junit4:junit4] 2> 657764 T1288 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 657764 T1288 oasc.SyncStrategy.syncToMe http://127.0.0.1:40299/c_qo/w/collection1/ has no replicas
[junit4:junit4] 2> 657764 T1288 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40299/c_qo/w/collection1/
[junit4:junit4] 2> 657764 T1288 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 658044 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 658056 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 658056 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 658056 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 658057 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 658070 T1288 oasc.ZkController.register We are http://127.0.0.1:40299/c_qo/w/collection1/ and leader is http://127.0.0.1:40299/c_qo/w/collection1/
[junit4:junit4] 2> 658070 T1288 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40299/c_qo/w
[junit4:junit4] 2> 658070 T1288 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 658070 T1288 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 658070 T1288 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 658073 T1288 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 658074 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 658074 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 658075 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 658147 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 658149 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33440
[junit4:junit4] 2> 658149 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 658150 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 658150 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211
[junit4:junit4] 2> 658150 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/solr.xml
[junit4:junit4] 2> 658150 T1229 oasc.CoreContainer.<init> New CoreContainer 1056476166
[junit4:junit4] 2> 658151 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/'
[junit4:junit4] 2> 658151 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/'
[junit4:junit4] 2> 658189 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 658190 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 658190 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 658190 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 658190 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 658191 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 658191 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 658191 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 658191 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 658192 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 658194 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 658195 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 658195 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 658196 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 658199 T1301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4db02d65 name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 658199 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 658201 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 658206 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 658208 T1303 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2df47d53 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 658208 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 658212 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 659215 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33440_c_qo%2Fw
[junit4:junit4] 2> 659217 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33440_c_qo%2Fw
[junit4:junit4] 2> 659222 T1258 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 659222 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659223 T1271 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 659223 T1303 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 659223 T1251 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 659223 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659224 T1287 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 659224 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659224 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659232 T1304 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 659233 T1304 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 659566 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 659567 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40299/c_qo/w",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:40299_c_qo%2Fw"}
[junit4:junit4] 2> 659573 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33440/c_qo/w",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:33440_c_qo%2Fw"}
[junit4:junit4] 2> 659574 T1252 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 659574 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 659580 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659580 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659581 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659581 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 659580 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 660235 T1304 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/collection1
[junit4:junit4] 2> 660235 T1304 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 660236 T1304 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 660236 T1304 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 660238 T1304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/collection1/'
[junit4:junit4] 2> 660239 T1304 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/collection1/lib/README' to classloader
[junit4:junit4] 2> 660240 T1304 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 660297 T1304 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 660326 T1304 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 660327 T1304 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 660331 T1304 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 660668 T1304 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 660673 T1304 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 660675 T1304 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 660687 T1304 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 660690 T1304 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 660694 T1304 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 660694 T1304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 660695 T1304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 660695 T1304 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 660696 T1304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 660696 T1304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 660696 T1304 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 660697 T1304 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371594427211/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty3/
[junit4:junit4] 2> 660697 T1304 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 660697 T1304 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 660698 T1304 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty3
[junit4:junit4] 2> 660698 T1304 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty3/index/
[junit4:junit4] 2> 660698 T1304 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 660698 T1304 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty3/index
[junit4:junit4] 2> 660701 T1304 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 660701 T1304 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 660703 T1304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 660703 T1304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 660704 T1304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 660704 T1304 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 660705 T1304 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 660705 T1304 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 660705 T1304 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 660706 T1304 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 660706 T1304 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 660708 T1304 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 660711 T1304 oass.SolrIndexSearcher.<init> Opening Searcher@f3ffcf3 main
[junit4:junit4] 2> 660711 T1304 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 660712 T1304 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 660723 T1305 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f3ffcf3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 660725 T1304 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 660725 T1304 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33440/c_qo/w collection:collection1 shard:shard1
[junit4:junit4] 2> 660726 T1304 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 660732 T1304 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 660735 T1304 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 660735 T1304 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 660735 T1304 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33440/c_qo/w/collection1/
[junit4:junit4] 2> 660735 T1304 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 660735 T1304 oasc.SyncStrategy.syncToMe http://127.0.0.1:33440/c_qo/w/collection1/ has no replicas
[junit4:junit4] 2> 660735 T1304 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33440/c_qo/w/collection1/
[junit4:junit4] 2> 660736 T1304 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 661093 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 661105 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 661105 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 661106 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 661106 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 661105 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 661141 T1304 oasc.ZkController.register We are http://127.0.0.1:33440/c_qo/w/collection1/ and leader is http://127.0.0.1:33440/c_qo/w/collection1/
[junit4:junit4] 2> 661141 T1304 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33440/c_qo/w
[junit4:junit4] 2> 661142 T1304 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 661142 T1304 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 661142 T1304 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 661145 T1304 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 661147 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 661148 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 661148 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 661231 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 661234 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52113
[junit4:junit4] 2> 661235 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 661235 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 661235 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289
[junit4:junit4] 2> 661235 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/solr.xml
[junit4:junit4] 2> 661236 T1229 oasc.CoreContainer.<init> New CoreContainer 841060181
[junit4:junit4] 2> 661236 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/'
[junit4:junit4] 2> 661236 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/'
[junit4:junit4] 2> 661274 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 661275 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 661275 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 661275 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 661275 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 661276 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 661276 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 661276 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 661277 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 661277 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 661280 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 661280 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 661280 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 661281 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 661283 T1317 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12cff4e name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 661283 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 661285 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 661288 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 661289 T1319 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c88a91 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 661290 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 661292 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 662295 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52113_c_qo%2Fw
[junit4:junit4] 2> 662297 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52113_c_qo%2Fw
[junit4:junit4] 2> 662300 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 662301 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 662302 T1258 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 662303 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662303 T1271 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 662303 T1319 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 662303 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662303 T1303 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 662305 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662306 T1251 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 662308 T1287 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 662315 T1320 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 662315 T1320 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 662616 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 662617 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33440/c_qo/w",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:33440_c_qo%2Fw"}
[junit4:junit4] 2> 662623 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52113/c_qo/w",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:52113_c_qo%2Fw"}
[junit4:junit4] 2> 662624 T1252 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 662624 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 662631 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662631 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662631 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662631 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662631 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 662631 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 663317 T1320 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/collection1
[junit4:junit4] 2> 663317 T1320 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 663318 T1320 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 663318 T1320 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 663320 T1320 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/collection1/'
[junit4:junit4] 2> 663321 T1320 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/collection1/lib/README' to classloader
[junit4:junit4] 2> 663321 T1320 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 663358 T1320 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 663386 T1320 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 663388 T1320 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 663391 T1320 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 663742 T1320 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 663747 T1320 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 663749 T1320 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 663760 T1320 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 663763 T1320 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 663766 T1320 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 663768 T1320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 663768 T1320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 663768 T1320 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 663769 T1320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 663769 T1320 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 663770 T1320 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 663770 T1320 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371594430289/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/
[junit4:junit4] 2> 663770 T1320 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 663770 T1320 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 663771 T1320 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4
[junit4:junit4] 2> 663771 T1320 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index/
[junit4:junit4] 2> 663771 T1320 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 663771 T1320 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index
[junit4:junit4] 2> 663775 T1320 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 663776 T1320 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 663777 T1320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 663778 T1320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 663778 T1320 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 663778 T1320 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 663779 T1320 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 663779 T1320 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 663779 T1320 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 663780 T1320 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 663780 T1320 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 663781 T1320 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 663792 T1320 oass.SolrIndexSearcher.<init> Opening Searcher@6d678630 main
[junit4:junit4] 2> 663793 T1320 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 663793 T1320 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 663796 T1321 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d678630 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 663798 T1320 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 663798 T1320 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52113/c_qo/w collection:collection1 shard:shard2
[junit4:junit4] 2> 663801 T1320 oasc.ZkController.register We are http://127.0.0.1:52113/c_qo/w/collection1/ and leader is http://127.0.0.1:46219/c_qo/w/collection1/
[junit4:junit4] 2> 663801 T1320 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52113/c_qo/w
[junit4:junit4] 2> 663801 T1320 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 663801 T1320 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1359 name=collection1 org.apache.solr.core.SolrCore@2b7b2368 url=http://127.0.0.1:52113/c_qo/w/collection1 node=127.0.0.1:52113_c_qo%2Fw C1359_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, base_url=http://127.0.0.1:52113/c_qo/w, state=down, node_name=127.0.0.1:52113_c_qo%2Fw}
[junit4:junit4] 2> 663801 T1322 C1359 P52113 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 663802 T1322 C1359 P52113 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 663802 T1320 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 663802 T1322 C1359 P52113 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 663802 T1322 C1359 P52113 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 663803 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 663803 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 663803 T1322 C1359 P52113 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 663803 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 663808 T1267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 663877 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 663879 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34374
[junit4:junit4] 2> 663880 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 663881 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 663881 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940
[junit4:junit4] 2> 663881 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/solr.xml
[junit4:junit4] 2> 663881 T1229 oasc.CoreContainer.<init> New CoreContainer 1557460862
[junit4:junit4] 2> 663882 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/'
[junit4:junit4] 2> 663882 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/'
[junit4:junit4] 2> 663921 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 663921 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 663922 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 663922 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 663922 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 663922 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 663923 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 663923 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 663923 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 663923 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 663926 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 663927 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 663927 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 663928 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 663929 T1334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fd4903b name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 663930 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 663931 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 663935 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 663937 T1336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23f0147e name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 663937 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 663940 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 664139 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 664141 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52113/c_qo/w",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:52113_c_qo%2Fw"}
[junit4:junit4] 2> 664147 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664148 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664148 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664148 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664148 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664148 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664147 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664808 T1267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 664809 T1267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={core=collection1&onlyIfLeader=true&nodeName=127.0.0.1:52113_c_qo%252Fw&wt=javabin&version=2&action=PREPRECOVERY&coreNodeName=5&checkLive=true&state=recovering} status=0 QTime=1001
[junit4:junit4] 2> 664943 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34374_c_qo%2Fw
[junit4:junit4] 2> 664946 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34374_c_qo%2Fw
[junit4:junit4] 2> 664950 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664950 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664950 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664950 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 664951 T1251 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664951 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 664951 T1258 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664952 T1319 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664952 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 664952 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 664954 T1336 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664955 T1303 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664955 T1287 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664956 T1271 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 664964 T1337 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 664964 T1337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 665657 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 665658 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34374/c_qo/w",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:34374_c_qo%2Fw"}
[junit4:junit4] 2> 665659 T1252 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 665659 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 665665 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665665 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665665 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665665 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665665 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665666 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665665 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 665966 T1337 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/collection1
[junit4:junit4] 2> 665966 T1337 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 665968 T1337 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 665968 T1337 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 665970 T1337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/collection1/'
[junit4:junit4] 2> 665971 T1337 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/collection1/lib/README' to classloader
[junit4:junit4] 2> 665971 T1337 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 666030 T1337 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 666058 T1337 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 666059 T1337 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 666063 T1337 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 666395 T1337 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 666400 T1337 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 666402 T1337 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 666414 T1337 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 666417 T1337 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 666420 T1337 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 666421 T1337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 666422 T1337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 666422 T1337 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 666423 T1337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 666423 T1337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 666423 T1337 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 666423 T1337 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371594432940/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/
[junit4:junit4] 2> 666424 T1337 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 666424 T1337 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 666424 T1337 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5
[junit4:junit4] 2> 666425 T1337 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index/
[junit4:junit4] 2> 666425 T1337 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 666425 T1337 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index
[junit4:junit4] 2> 666428 T1337 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 666428 T1337 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 666430 T1337 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 666430 T1337 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 666431 T1337 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 666431 T1337 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 666432 T1337 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 666432 T1337 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 666432 T1337 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 666433 T1337 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 666433 T1337 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 666435 T1337 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 666445 T1337 oass.SolrIndexSearcher.<init> Opening Searcher@35932d22 main
[junit4:junit4] 2> 666446 T1337 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 666446 T1337 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 666451 T1338 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@35932d22 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 666453 T1337 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 666453 T1337 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34374/c_qo/w collection:collection1 shard:shard3
[junit4:junit4] 2> 666456 T1337 oasc.ZkController.register We are http://127.0.0.1:34374/c_qo/w/collection1/ and leader is http://127.0.0.1:40299/c_qo/w/collection1/
[junit4:junit4] 2> 666456 T1337 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34374/c_qo/w
[junit4:junit4] 2> 666456 T1337 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 666456 T1337 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1360 name=collection1 org.apache.solr.core.SolrCore@543e0e9 url=http://127.0.0.1:34374/c_qo/w/collection1 node=127.0.0.1:34374_c_qo%2Fw C1360_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, base_url=http://127.0.0.1:34374/c_qo/w, state=down, node_name=127.0.0.1:34374_c_qo%2Fw}
[junit4:junit4] 2> 666457 T1339 C1360 P34374 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 666457 T1337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 666457 T1339 C1360 P34374 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 666458 T1339 C1360 P34374 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 666458 T1339 C1360 P34374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 666458 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 666459 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 666459 T1339 C1360 P34374 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 666459 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 666464 T1281 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 666543 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 666545 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33732
[junit4:junit4] 2> 666546 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 666546 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 666546 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596
[junit4:junit4] 2> 666547 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/solr.xml
[junit4:junit4] 2> 666547 T1229 oasc.CoreContainer.<init> New CoreContainer 794667140
[junit4:junit4] 2> 666547 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/'
[junit4:junit4] 2> 666547 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/'
[junit4:junit4] 2> 666585 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 666586 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 666586 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 666586 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 666586 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 666587 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 666587 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 666587 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 666587 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 666588 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 666591 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 666591 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 666591 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 666592 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 666594 T1351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cafe174 name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 666594 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 666597 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 666600 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 666602 T1353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a1cf103 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 666602 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 666606 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1361 name=collection1 org.apache.solr.core.SolrCore@2b7b2368 url=http://127.0.0.1:52113/c_qo/w/collection1 node=127.0.0.1:52113_c_qo%2Fw C1361_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard2, base_url=http://127.0.0.1:52113/c_qo/w, state=recovering, node_name=127.0.0.1:52113_c_qo%2Fw}
[junit4:junit4] 2> 666809 T1322 C1361 P52113 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46219/c_qo/w/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 666810 T1322 C1361 P52113 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52113/c_qo/w START replicas=[http://127.0.0.1:46219/c_qo/w/collection1/] nUpdates=100
[junit4:junit4] 2> 666810 T1322 C1361 P52113 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 666811 T1322 C1361 P52113 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 666811 T1322 C1361 P52113 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 666811 T1322 C1361 P52113 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 666811 T1322 C1361 P52113 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 666811 T1322 C1361 P52113 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46219/c_qo/w/collection1/. core=collection1
[junit4:junit4] 2> 666812 T1322 C1361 P52113 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1362 name=collection1 org.apache.solr.core.SolrCore@7d457a8f url=http://127.0.0.1:46219/c_qo/w/collection1 node=127.0.0.1:46219_c_qo%2Fw C1362_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard2, base_url=http://127.0.0.1:46219/c_qo/w, state=active, node_name=127.0.0.1:46219_c_qo%2Fw, leader=true}
[junit4:junit4] 2> 666817 T1265 C1362 P46219 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/get params={getVersions=100&wt=javabin&version=2&distrib=false&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 666821 T1266 C1362 P46219 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 666823 T1266 C1362 P46219 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 666824 T1266 C1362 P46219 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 666836 T1266 C1362 P46219 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 666836 T1266 C1362 P46219 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 666837 T1266 C1362 P46219 oass.SolrIndexSearcher.<init> Opening Searcher@655be3fc realtime
[junit4:junit4] 2> 666837 T1266 C1362 P46219 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 666837 T1266 C1362 P46219 oasup.LogUpdateProcessor.finish [collection1] webapp=/c_qo/w path=/update params={commit_end_point=true&wt=javabin&version=2&softCommit=false&commit=true&waitSearcher=true&openSearcher=false} {commit=} 0 16
[junit4:junit4] 2> 666838 T1322 C1361 P52113 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 666838 T1322 C1361 P52113 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> ASYNC NEW_CORE C1363 name=collection1 org.apache.solr.core.SolrCore@7d457a8f url=http://127.0.0.1:46219/c_qo/w/collection1 node=127.0.0.1:46219_c_qo%2Fw C1363_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard2, base_url=http://127.0.0.1:46219/c_qo/w, state=active, node_name=127.0.0.1:46219_c_qo%2Fw, leader=true}
[junit4:junit4] 2> 666852 T1264 C1363 P46219 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 666853 T1264 C1363 P46219 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/replication params={wt=javabin&version=2&command=indexversion&qt=/replication} status=0 QTime=14
[junit4:junit4] 2> ASYNC NEW_CORE C1364 name=collection1 org.apache.solr.core.SolrCore@2b7b2368 url=http://127.0.0.1:52113/c_qo/w/collection1 node=127.0.0.1:52113_c_qo%2Fw C1364_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard2, base_url=http://127.0.0.1:52113/c_qo/w, state=recovering, node_name=127.0.0.1:52113_c_qo%2Fw}
[junit4:junit4] 2> 666853 T1322 C1364 P52113 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 666854 T1322 C1364 P52113 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 666854 T1322 C1364 P52113 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 666856 T1265 C1363 P46219 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/replication params={wt=javabin&version=2&command=filelist&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 666856 T1322 C1364 P52113 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 666857 T1322 C1364 P52113 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index.20130619102715989
[junit4:junit4] 2> 666857 T1322 C1364 P52113 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index.20130619102715989 lockFactory=org.apache.lucene.store.NativeFSLockFactory@361e7614 fullCopy=false
[junit4:junit4] 2> 666860 T1266 C1363 P46219 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/replication params={wt=filestream&checksum=true&command=filecontent&file=segments_2&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 666861 T1322 C1364 P52113 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 666866 T1322 C1364 P52113 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 666867 T1322 C1364 P52113 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 666869 T1322 C1364 P52113 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 666869 T1322 C1364 P52113 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 666869 T1322 C1364 P52113 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 666869 T1322 C1364 P52113 oass.SolrIndexSearcher.<init> Opening Searcher@602c1eac main
[junit4:junit4] 2> 666870 T1321 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@602c1eac main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 666871 T1322 C1364 P52113 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index.20130619102715989 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index.20130619102715989;done=true>>]
[junit4:junit4] 2> 666871 T1322 C1364 P52113 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index.20130619102715989
[junit4:junit4] 2> 666871 T1322 C1364 P52113 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty4/index.20130619102715989
[junit4:junit4] 2> 666872 T1322 C1364 P52113 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 666872 T1322 C1364 P52113 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 666872 T1322 C1364 P52113 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 666872 T1322 C1364 P52113 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 666874 T1322 C1364 P52113 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 667174 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 667176 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34374/c_qo/w",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:34374_c_qo%2Fw"}
[junit4:junit4] 2> 667180 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52113/c_qo/w",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:52113_c_qo%2Fw"}
[junit4:junit4] 2> 667183 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667183 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667465 T1281 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 667465 T1281 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={core=collection1&onlyIfLeader=true&nodeName=127.0.0.1:34374_c_qo%252Fw&wt=javabin&version=2&action=PREPRECOVERY&coreNodeName=6&checkLive=true&state=recovering} status=0 QTime=1001
[junit4:junit4] 2> 667609 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33732_c_qo%2Fw
[junit4:junit4] 2> 667626 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33732_c_qo%2Fw
[junit4:junit4] 2> 667630 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667630 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667630 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667630 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667631 T1258 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667631 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 667630 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667631 T1251 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667630 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 667631 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 667632 T1271 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667633 T1353 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667633 T1287 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667633 T1303 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667634 T1319 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667634 T1336 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 667637 T1356 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 667637 T1356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 668690 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 668691 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33732/c_qo/w",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:33732_c_qo%2Fw"}
[junit4:junit4] 2> 668691 T1252 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 668691 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 668695 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668695 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668696 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668695 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668695 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668696 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668696 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 668695 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> ASYNC NEW_CORE C1365 name=collection1 org.apache.solr.core.SolrCore@543e0e9 url=http://127.0.0.1:34374/c_qo/w/collection1 node=127.0.0.1:34374_c_qo%2Fw C1365_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard3, base_url=http://127.0.0.1:34374/c_qo/w, state=recovering, node_name=127.0.0.1:34374_c_qo%2Fw}
[junit4:junit4] 2> 669466 T1339 C1365 P34374 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40299/c_qo/w/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 669467 T1339 C1365 P34374 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:34374/c_qo/w START replicas=[http://127.0.0.1:40299/c_qo/w/collection1/] nUpdates=100
[junit4:junit4] 2> 669467 T1339 C1365 P34374 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 669468 T1339 C1365 P34374 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 669468 T1339 C1365 P34374 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 669468 T1339 C1365 P34374 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 669468 T1339 C1365 P34374 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 669469 T1339 C1365 P34374 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40299/c_qo/w/collection1/. core=collection1
[junit4:junit4] 2> 669469 T1339 C1365 P34374 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1366 name=collection1 org.apache.solr.core.SolrCore@2dddb4ae url=http://127.0.0.1:40299/c_qo/w/collection1 node=127.0.0.1:40299_c_qo%2Fw C1366_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard3, base_url=http://127.0.0.1:40299/c_qo/w, state=active, node_name=127.0.0.1:40299_c_qo%2Fw, leader=true}
[junit4:junit4] 2> 669472 T1281 C1366 P40299 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/get params={getVersions=100&wt=javabin&version=2&distrib=false&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 669473 T1282 C1366 P40299 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 669475 T1282 C1366 P40299 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 669475 T1282 C1366 P40299 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 669486 T1282 C1366 P40299 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 669486 T1282 C1366 P40299 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 669486 T1282 C1366 P40299 oass.SolrIndexSearcher.<init> Opening Searcher@230f4e6e realtime
[junit4:junit4] 2> 669487 T1282 C1366 P40299 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 669487 T1282 C1366 P40299 oasup.LogUpdateProcessor.finish [collection1] webapp=/c_qo/w path=/update params={commit_end_point=true&wt=javabin&version=2&softCommit=false&commit=true&waitSearcher=true&openSearcher=false} {commit=} 0 14
[junit4:junit4] 2> 669487 T1339 C1365 P34374 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 669488 T1339 C1365 P34374 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 669489 T1280 C1366 P40299 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 669489 T1280 C1366 P40299 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/replication params={wt=javabin&version=2&command=indexversion&qt=/replication} status=0 QTime=0
[junit4:junit4] 2> 669490 T1339 C1365 P34374 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 669490 T1339 C1365 P34374 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 669490 T1339 C1365 P34374 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 669492 T1281 C1366 P40299 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/replication params={wt=javabin&version=2&command=filelist&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 669492 T1339 C1365 P34374 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 669493 T1339 C1365 P34374 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index.20130619102718626
[junit4:junit4] 2> 669493 T1339 C1365 P34374 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index.20130619102718626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2615b010 fullCopy=false
[junit4:junit4] 2> 669495 T1282 C1366 P40299 oasc.SolrCore.execute [collection1] webapp=/c_qo/w path=/replication params={wt=filestream&checksum=true&command=filecontent&file=segments_2&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 669496 T1339 C1365 P34374 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 669506 T1339 C1365 P34374 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 669506 T1339 C1365 P34374 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 669508 T1339 C1365 P34374 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 669509 T1339 C1365 P34374 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 669509 T1339 C1365 P34374 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 669509 T1339 C1365 P34374 oass.SolrIndexSearcher.<init> Opening Searcher@610b4a9a main
[junit4:junit4] 2> 669510 T1338 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@610b4a9a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 669510 T1339 C1365 P34374 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index.20130619102718626 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index.20130619102718626;done=true>>]
[junit4:junit4] 2> 669510 T1339 C1365 P34374 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index.20130619102718626
[junit4:junit4] 2> 669510 T1339 C1365 P34374 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty5/index.20130619102718626
[junit4:junit4] 2> 669511 T1339 C1365 P34374 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 669511 T1339 C1365 P34374 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 669511 T1339 C1365 P34374 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 669511 T1339 C1365 P34374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 669526 T1339 C1365 P34374 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 669639 T1356 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/collection1
[junit4:junit4] 2> 669639 T1356 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 669640 T1356 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 669640 T1356 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 669643 T1356 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/collection1/'
[junit4:junit4] 2> 669644 T1356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/collection1/lib/README' to classloader
[junit4:junit4] 2> 669644 T1356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 669685 T1356 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 669720 T1356 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 669721 T1356 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 669724 T1356 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 670036 T1356 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 670041 T1356 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 670042 T1356 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 670055 T1356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 670058 T1356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 670061 T1356 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 670062 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 670062 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 670062 T1356 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 670063 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 670063 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 670064 T1356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 670064 T1356 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371594435596/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty6/
[junit4:junit4] 2> 670064 T1356 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 670065 T1356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 670065 T1356 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty6
[junit4:junit4] 2> 670065 T1356 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty6/index/
[junit4:junit4] 2> 670065 T1356 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 670066 T1356 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty6/index
[junit4:junit4] 2> 670070 T1356 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 670071 T1356 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 670074 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 670074 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 670075 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 670076 T1356 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 670077 T1356 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 670077 T1356 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 670079 T1356 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 670080 T1356 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 670081 T1356 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 670083 T1356 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 670088 T1356 oass.SolrIndexSearcher.<init> Opening Searcher@5f4b27d3 main
[junit4:junit4] 2> 670089 T1356 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 670089 T1356 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 670096 T1359 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f4b27d3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 670098 T1356 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 670099 T1356 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33732/c_qo/w collection:collection1 shard:shard1
[junit4:junit4] 2> 670104 T1356 oasc.ZkController.register We are http://127.0.0.1:33732/c_qo/w/collection1/ and leader is http://127.0.0.1:33440/c_qo/w/collection1/
[junit4:junit4] 2> 670105 T1356 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33732/c_qo/w
[junit4:junit4] 2> 670105 T1356 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 670105 T1356 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1367 name=collection1 org.apache.solr.core.SolrCore@3238b24e url=http://127.0.0.1:33732/c_qo/w/collection1 node=127.0.0.1:33732_c_qo%2Fw C1367_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, base_url=http://127.0.0.1:33732/c_qo/w, state=down, node_name=127.0.0.1:33732_c_qo%2Fw}
[junit4:junit4] 2> 670106 T1360 C1367 P33732 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 670106 T1360 C1367 P33732 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 670107 T1360 C1367 P33732 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 670107 T1356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 670107 T1360 C1367 P33732 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 670109 T1360 C1367 P33732 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 670109 T1229 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 670110 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 670110 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 670116 T1296 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 670184 T1229 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 670187 T1229 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50422
[junit4:junit4] 2> 670188 T1229 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 670188 T1229 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 670188 T1229 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248
[junit4:junit4] 2> 670188 T1229 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/solr.xml
[junit4:junit4] 2> 670189 T1229 oasc.CoreContainer.<init> New CoreContainer 1554652775
[junit4:junit4] 2> 670189 T1229 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/'
[junit4:junit4] 2> 670189 T1229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/'
[junit4:junit4] 2> 670200 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 670201 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34374/c_qo/w",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:34374_c_qo%2Fw"}
[junit4:junit4] 2> 670205 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33732/c_qo/w",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:33732_c_qo%2Fw"}
[junit4:junit4] 2> 670211 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670211 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 670244 T1229 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 670244 T1229 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 670245 T1229 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 670245 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 670245 T1229 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 670245 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 670245 T1229 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 670246 T1229 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 670246 T1229 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 670246 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 670249 T1229 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 670249 T1229 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56626/solr
[junit4:junit4] 2> 670249 T1229 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 670250 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 670252 T1372 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1af68516 name:ZooKeeperConnection Watcher:127.0.0.1:56626 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 670252 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 670254 T1229 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 670257 T1229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 670258 T1374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ff8f528 name:ZooKeeperConnection Watcher:127.0.0.1:56626/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 670259 T1229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 670263 T1229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 671117 T1296 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 671117 T1296 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={core=collection1&onlyIfLeader=true&nodeName=127.0.0.1:33732_c_qo%252Fw&wt=javabin&version=2&action=PREPRECOVERY&coreNodeName=7&checkLive=true&state=recovering} status=0 QTime=1001
[junit4:junit4] 2> 671266 T1229 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50422_c_qo%2Fw
[junit4:junit4] 2> 671287 T1229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50422_c_qo%2Fw
[junit4:junit4] 2> 671308 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 671308 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 671309 T1353 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671309 T1336 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671310 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671309 T1271 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671311 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671309 T1303 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671310 T1374 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671310 T1258 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671310 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671310 T1251 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671312 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671312 T1287 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671313 T1319 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 671312 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671311 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671333 T1375 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 671333 T1375 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 671735 T1252 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 671736 T1252 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50422/c_qo/w",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "node_name":"127.0.0.1:50422_c_qo%2Fw"}
[junit4:junit4] 2> 671736 T1252 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 671736 T1252 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 671741 T1251 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1319 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1258 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 671741 T1287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 672347 T1375 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/collection1
[junit4:junit4] 2> 672348 T1375 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 672349 T1375 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 672349 T1375 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 672351 T1375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/collection1/'
[junit4:junit4] 2> 672352 T1375 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/collection1/lib/README' to classloader
[junit4:junit4] 2> 672353 T1375 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 672392 T1375 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 672435 T1375 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 672437 T1375 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 672441 T1375 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 672775 T1375 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 672779 T1375 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 672781 T1375 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 672800 T1375 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 672803 T1375 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 672808 T1375 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 672809 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 672810 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 672810 T1375 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 672812 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 672812 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 672813 T1375 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 672813 T1375 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371594439248/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty7/
[junit4:junit4] 2> 672813 T1375 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2983f773
[junit4:junit4] 2> 672814 T1375 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 672815 T1375 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty7
[junit4:junit4] 2> 672816 T1375 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty7/index/
[junit4:junit4] 2> 672816 T1375 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 672817 T1375 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty7/index
[junit4:junit4] 2> 672821 T1375 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 672822 T1375 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 672825 T1375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 672825 T1375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 672826 T1375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 672827 T1375 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 672827 T1375 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 672828 T1375 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 672828 T1375 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 672828 T1375 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 672829 T1375 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 672830 T1375 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 672832 T1375 oass.SolrIndexSearcher.<init> Opening Searcher@1e1ac214 main
[junit4:junit4] 2> 672833 T1375 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 672833 T1375 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 672839 T1376 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e1ac214 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 672841 T1375 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 672842 T1375 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50422/c_qo/w collection:collection1 shard:shard2
[junit4:junit4] 2> 672847 T1375 oasc.ZkController.register We are http://127.0.0.1:50422/c_qo/w/collection1/ and leader is http://127.0.0.1:46219/c_qo/w/collection1/
[junit4:junit4] 2> 672847 T1375 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50422/c_qo/w
[junit4:junit4] 2> 672847 T1375 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 672848 T1375 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1368 name=collection1 org.apache.solr.core.SolrCore@3922f37 url=http://127.0.0.1:50422/c_qo/w/collection1 node=127.0.0.1:50422_c_qo%2Fw C1368_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, base_url=http://127.0.0.1:50422/c_qo/w, state=down, node_name=127.0.0.1:50422_c_qo%2Fw}
[junit4:junit4] 2> 672848 T1377 C1368 P50422 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 672849 T1377 C1368 P50422 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 672849 T1375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 672849 T1377 C1368 P50422 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4
[...truncated too long message...]
opped o.e.j.s.ServletContextHandler{/c_qo/w,null}
[junit4:junit4] 2> 790590 T1229 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50639
[junit4:junit4] 2> 790590 T1229 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1986538706
[junit4:junit4] 2> 791592 T1229 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 791593 T1229 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 791596 T1229 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@ae93e39
[junit4:junit4] 2> 791597 T1229 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=59,cumulative_deletesById=29,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 791598 T1229 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 791598 T1229 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 791599 T1229 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> ASYNC NEW_CORE C1598 name=collection1 org.apache.solr.core.SolrCore@ae93e39 url=http://127.0.0.1:50639/c_qo/w/collection1 node=127.0.0.1:50639_c_qo%2Fw C1598_STATE=coll:collection1 core:collection1 props:{core=collection1, collection=collection1, shard=shard1, base_url=http://127.0.0.1:50639/c_qo/w, state=active, node_name=127.0.0.1:50639_c_qo%2Fw, leader=true}
[junit4:junit4] 2> 791617 T1229 C1598 P50639 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12/index,segFN=segments_q,generation=26}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12/index,segFN=segments_r,generation=27}
[junit4:junit4] 2> 791618 T1229 C1598 P50639 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 27
[junit4:junit4] 2> 791620 T1229 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 791621 T1229 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 791621 T1229 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12;done=false>>]
[junit4:junit4] 2> 791622 T1229 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12
[junit4:junit4] 2> 791622 T1229 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12/index;done=false>>]
[junit4:junit4] 2> 791623 T1229 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371594417647/jetty12/index
[junit4:junit4] 2> 791624 T1706 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89888811755110428-127.0.0.1:50639_c_qo%2Fw-n_0000000012) am no longer a leader.
[junit4:junit4] 2> 791627 T1467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 791627 T1467 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 791628 T1467 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 791650 T1229 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/c_qo/w,null}
[junit4:junit4] 2> 791735 T1229 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 791737 T1229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56626 56626
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=F73BF3A867A3596E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=Pacific/Majuro -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 143s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 30 from http://127.0.0.1:34724/c_qo/w/collection1lastClient and got 29 from http://127.0.0.1:50639/c_qo/w/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([F73BF3A867A3596E:76DD7DB010FC3952]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 791798 T1229 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 143293 T1228 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 791815 T1543 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 791815 T1543 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 791815 T1543 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=fi_FI, timezone=Pacific/Majuro
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=3,free=21968664,total=149848064
[junit4:junit4] 2> NOTE: All tests run in this JVM: [PingRequestHandlerTest, TestFiltering, HighlighterTest, SpellCheckComponentTest, DirectSolrSpellCheckerTest, TestUpdate, XmlUpdateRequestHandlerTest, TestArbitraryIndexDir, TestRecovery, TestAnalyzedSuggestions, TestIndexingPerformance, DirectUpdateHandlerTest, TestSolrXml, ChaosMonkeyShardSplitTest, TestReload, CoreContainerCoreInitFailuresTest, SignatureUpdateProcessorFactoryTest, SpellPossibilityIteratorTest, TestSchemaVersionResource, DOMUtilTest, SolrRequestParserTest, OpenExchangeRatesOrgProviderTest, HardAutoCommitTest, TestCollationField, SolrPluginUtilsTest, TestCoreDiscovery, UpdateParamsTest, TestSystemIdResolver, TestPropInject, OutputWriterTest, TermVectorComponentDistributedTest, DistanceFunctionTest, ConvertedLegacyTest, TestBM25SimilarityFactory, TestLuceneMatchVersion, TestFoldingMultitermQuery, TestWordDelimiterFilterFactory, TestFieldResource, IndexReaderFactoryTest, NumericFieldsTest, DebugComponentTest, TestDynamicFieldResource, TestQuerySenderNoQuery, IndexBasedSpellCheckerTest, URLClassifyProcessorTest, TestBinaryField, LeaderElectionIntegrationTest, UniqFieldsUpdateProcessorFactoryTest, TestIndexSearcher, TestPostingsSolrHighlighter, TestSolrXMLSerializer, RequiredFieldsTest, TestClassNameShortening, TestCloudManagedSchema, ZkNodePropsTest, PathHierarchyTokenizerFactoryTest, ShardRoutingCustomTest, StatsComponentTest, TestMultiCoreConfBootstrap, ShardSplitTest, TestLazyCores, SimplePostToolTest, SuggesterWFSTTest, DisMaxRequestHandlerTest, TestFastOutputStream, TestBinaryResponseWriter, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 143.56s, 1 test, 1 failure <<< FAILURES!
[...truncated 451 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1253 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 37 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b93 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6128 - Still Failing!
Posted by Robert Muir <rc...@gmail.com>.
this is an ugly test bug: i opened an issue for it (
https://issues.apache.org/jira/browse/LUCENE-5066)
I think the patch is ok to fix this test fail, but as noted the issue, we
should probably add a simple low-level test for this to all real
directories.
also, i think it would be good to move
DocumentsWriterPerThread.MAX_TERM_LENGTH_UTF8 somewhere else (e.g.
IndexWriter), so we can add a similar assert to all the codecs: This way
instead of OOM we know stuff is really jacked up.
On Tue, Jun 18, 2013 at 5:13 PM, Robert Muir <rc...@gmail.com> wrote:
> somehow this relates to the delegation in faultyindexinput (NOT its
> exception-generation, which isnt yet happening....)
>
> and the surrogates dance is somehow tickling the delegator-bug in such a
> way that it thinks a single term is 262MB. thats why it only fails on 3.x
>
>
> On Tue, Jun 18, 2013 at 4:38 PM, Robert Muir <rc...@gmail.com> wrote:
>
>> This has failed several times, since LUCENE-5038 was committed. But that
>> commit didn't really change the test, except to setCFSRatio(0.0)
>>
>> and this test indexes only one document!!!!!!!!!!
>>
>> I'll look at this later tonight if nobody beats me to it, seems like
>> something is really wrong.
>>
>> On Tue, Jun 18, 2013 at 4:21 PM, Policeman Jenkins Server <
>> jenkins@thetaphi.de> wrote:
>>
>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6128/
>>> Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
>>>
>>> 2 tests failed.
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.lucene.index.TestFieldsReader
>>>
>>> Error Message:
>>> Captured an uncaught exception in thread: Thread[id=298, name=Lucene
>>> Merge Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader]
>>>
>>> Stack Trace:
>>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>>> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0,
>>> state=RUNNABLE, group=TGRP-TestFieldsReader]
>>> Caused by: org.apache.lucene.index.MergePolicy$MergeException:
>>> java.lang.OutOfMemoryError: Java heap space
>>> at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
>>> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>> at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
>>> at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
>>> at
>>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
>>> at
>>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
>>> at
>>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
>>> at
>>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
>>> at
>>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
>>> at
>>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
>>> at
>>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>>> at
>>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
>>> at
>>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
>>> at
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
>>> at
>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
>>> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>>
>>>
>>> REGRESSION: org.apache.lucene.index.TestFieldsReader.testExceptions
>>>
>>> Error Message:
>>> this writer hit an OutOfMemoryError; cannot complete forceMerge
>>>
>>> Stack Trace:
>>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError;
>>> cannot complete forceMerge
>>> at
>>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
>>> at
>>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
>>> at
>>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
>>> at
>>> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:491)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>>> at
>>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>> at
>>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>>> at
>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>> at
>>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>>> at
>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>>> at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>>> at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>>> at
>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>> at
>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>>> at
>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>>
>>>
>>>
>>> Build Log:
>>> [...truncated 395 lines...]
>>> [junit4:junit4] Suite: org.apache.lucene.index.TestFieldsReader
>>> [junit4:junit4] 2> NOTE: reproduce with: ant test
>>> -Dtestcase=TestFieldsReader -Dtests.method=testExceptions
>>> -Dtests.seed=2EE02ABE17F63E4B -Dtests.multiplier=3 -Dtests.slow=true
>>> -Dtests.locale=ar_LB -Dtests.timezone=America/Boa_Vista
>>> -Dtests.file.encoding=UTF-8
>>> [junit4:junit4] ERROR 1.28s J0 | TestFieldsReader.testExceptions <<<
>>> [junit4:junit4] > Throwable #1: java.lang.IllegalStateException: this
>>> writer hit an OutOfMemoryError; cannot complete forceMerge
>>> [junit4:junit4] > at
>>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
>>> [junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
>>> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>>> [junit4:junit4] 2> WARNING: Will linger awaiting termination of 1
>>> leaked thread(s).
>>> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>>> uncaughtException
>>> [junit4:junit4] 2> WARNING: Uncaught exception in thread:
>>> Thread[Lucene Merge Thread #0,6,TGRP-TestFieldsReader]
>>> [junit4:junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
>>> java.lang.OutOfMemoryError: Java heap space
>>> [junit4:junit4] 2> at
>>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>> [junit4:junit4] 2> Caused by: java.lang.OutOfMemoryError: Java heap
>>> space
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>> [junit4:junit4] 2> at
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>> [junit4:junit4] 2>
>>> [junit4:junit4] 2> NOTE: test params are: codec=Lucene3x,
>>> sim=DefaultSimilarity, locale=ar_LB, timezone=America/Boa_Vista
>>> [junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle
>>> Corporation 1.8.0-ea
>>> (32-bit)/cpus=8,threads=1,free=505999600,total=520093696
>>> [junit4:junit4] 2> NOTE: All tests run in this JVM:
>>> [TestLucene40PostingsFormat, TestSimilarity2, Test2BPostingsBytes,
>>> TestDemo, TestStressAdvance, TestIndexWriterForceMerge, TestArrayUtil,
>>> TestDocTermOrds, TestSentinelIntSet, TestFieldCacheSanityChecker,
>>> TestDateTools, TestIndexCommit, TestLucene40PostingsReader,
>>> TestCachingCollector, TestShardSearching, TestStressNRT,
>>> TestNoMergeScheduler, TestTopFieldCollector, TestIntroSorter,
>>> TestNRTThreads, TestCompressingTermVectorsFormat, TestTermInfosReaderIndex,
>>> TestFastDecompressionMode, TestPayloadTermQuery, TestFieldsReader]
>>> [junit4:junit4] 2> NOTE: reproduce with: ant test
>>> -Dtestcase=TestFieldsReader -Dtests.seed=2EE02ABE17F63E4B
>>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LB
>>> -Dtests.timezone=America/Boa_Vista -Dtests.file.encoding=UTF-8
>>> [junit4:junit4] ERROR 0.00s J0 | TestFieldsReader (suite) <<<
>>> [junit4:junit4] > Throwable #1:
>>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>>> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0,
>>> state=RUNNABLE, group=TGRP-TestFieldsReader]
>>> [junit4:junit4] > Caused by:
>>> org.apache.lucene.index.MergePolicy$MergeException:
>>> java.lang.OutOfMemoryError: Java heap space
>>> [junit4:junit4] > at
>>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>>> [junit4:junit4] > Caused by: java.lang.OutOfMemoryError: Java heap
>>> space
>>> [junit4:junit4] > at
>>> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
>>> [junit4:junit4] > at
>>> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
>>> [junit4:junit4] > at
>>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
>>> [junit4:junit4] > at
>>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
>>> [junit4:junit4] > at
>>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
>>> [junit4:junit4] > at
>>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
>>> [junit4:junit4] > at
>>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>>> [junit4:junit4] > at
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>> [junit4:junit4] Completed on J0 in 1.73s, 2 tests, 2 errors <<< FAILURES!
>>>
>>> [...truncated 984 lines...]
>>> BUILD FAILED
>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The
>>> following error occurred while executing this line:
>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The
>>> following error occurred while executing this line:
>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The
>>> following error occurred while executing this line:
>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49:
>>> The following error occurred while executing this line:
>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246:
>>> The following error occurred while executing this line:
>>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890:
>>> There were test failures: 364 suites, 2312 tests, 1 suite-level error, 1
>>> error, 47 ignored (34 assumptions)
>>>
>>> Total time: 9 minutes 17 seconds
>>> Build step 'Invoke Ant' marked build as failure
>>> Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
>>> Archiving artifacts
>>> Recording test results
>>> Email was triggered for: Failure
>>> Sending email for trigger: Failure
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6128 - Still Failing!
Posted by Robert Muir <rc...@gmail.com>.
somehow this relates to the delegation in faultyindexinput (NOT its
exception-generation, which isnt yet happening....)
and the surrogates dance is somehow tickling the delegator-bug in such a
way that it thinks a single term is 262MB. thats why it only fails on 3.x
On Tue, Jun 18, 2013 at 4:38 PM, Robert Muir <rc...@gmail.com> wrote:
> This has failed several times, since LUCENE-5038 was committed. But that
> commit didn't really change the test, except to setCFSRatio(0.0)
>
> and this test indexes only one document!!!!!!!!!!
>
> I'll look at this later tonight if nobody beats me to it, seems like
> something is really wrong.
>
> On Tue, Jun 18, 2013 at 4:21 PM, Policeman Jenkins Server <
> jenkins@thetaphi.de> wrote:
>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6128/
>> Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
>>
>> 2 tests failed.
>> FAILED:
>> junit.framework.TestSuite.org.apache.lucene.index.TestFieldsReader
>>
>> Error Message:
>> Captured an uncaught exception in thread: Thread[id=298, name=Lucene
>> Merge Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader]
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0,
>> state=RUNNABLE, group=TGRP-TestFieldsReader]
>> Caused by: org.apache.lucene.index.MergePolicy$MergeException:
>> java.lang.OutOfMemoryError: Java heap space
>> at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
>> at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
>> at
>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
>> at
>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
>> at
>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
>> at
>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
>> at
>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
>> at
>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
>> at
>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>> at
>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
>> at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
>> at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
>> at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>>
>>
>> REGRESSION: org.apache.lucene.index.TestFieldsReader.testExceptions
>>
>> Error Message:
>> this writer hit an OutOfMemoryError; cannot complete forceMerge
>>
>> Stack Trace:
>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError;
>> cannot complete forceMerge
>> at
>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
>> at
>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
>> at
>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
>> at
>> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:491)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>> at
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>> at
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>> at
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> at
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> at java.lang.Thread.run(Thread.java:724)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 395 lines...]
>> [junit4:junit4] Suite: org.apache.lucene.index.TestFieldsReader
>> [junit4:junit4] 2> NOTE: reproduce with: ant test
>> -Dtestcase=TestFieldsReader -Dtests.method=testExceptions
>> -Dtests.seed=2EE02ABE17F63E4B -Dtests.multiplier=3 -Dtests.slow=true
>> -Dtests.locale=ar_LB -Dtests.timezone=America/Boa_Vista
>> -Dtests.file.encoding=UTF-8
>> [junit4:junit4] ERROR 1.28s J0 | TestFieldsReader.testExceptions <<<
>> [junit4:junit4] > Throwable #1: java.lang.IllegalStateException: this
>> writer hit an OutOfMemoryError; cannot complete forceMerge
>> [junit4:junit4] > at
>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
>> [junit4:junit4] > at
>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
>> [junit4:junit4] > at
>> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
>> [junit4:junit4] > at
>> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
>> [junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
>> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM
>> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>> [junit4:junit4] 2> WARNING: Will linger awaiting termination of 1
>> leaked thread(s).
>> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM
>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>> uncaughtException
>> [junit4:junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene
>> Merge Thread #0,6,TGRP-TestFieldsReader]
>> [junit4:junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
>> java.lang.OutOfMemoryError: Java heap space
>> [junit4:junit4] 2> at
>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> [junit4:junit4] 2> Caused by: java.lang.OutOfMemoryError: Java heap
>> space
>> [junit4:junit4] 2> at
>> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
>> [junit4:junit4] 2> at
>> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
>> [junit4:junit4] 2> at
>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
>> [junit4:junit4] 2> at
>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
>> [junit4:junit4] 2> at
>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
>> [junit4:junit4] 2> at
>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
>> [junit4:junit4] 2> at
>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> [junit4:junit4] 2> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> NOTE: test params are: codec=Lucene3x,
>> sim=DefaultSimilarity, locale=ar_LB, timezone=America/Boa_Vista
>> [junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation
>> 1.8.0-ea (32-bit)/cpus=8,threads=1,free=505999600,total=520093696
>> [junit4:junit4] 2> NOTE: All tests run in this JVM:
>> [TestLucene40PostingsFormat, TestSimilarity2, Test2BPostingsBytes,
>> TestDemo, TestStressAdvance, TestIndexWriterForceMerge, TestArrayUtil,
>> TestDocTermOrds, TestSentinelIntSet, TestFieldCacheSanityChecker,
>> TestDateTools, TestIndexCommit, TestLucene40PostingsReader,
>> TestCachingCollector, TestShardSearching, TestStressNRT,
>> TestNoMergeScheduler, TestTopFieldCollector, TestIntroSorter,
>> TestNRTThreads, TestCompressingTermVectorsFormat, TestTermInfosReaderIndex,
>> TestFastDecompressionMode, TestPayloadTermQuery, TestFieldsReader]
>> [junit4:junit4] 2> NOTE: reproduce with: ant test
>> -Dtestcase=TestFieldsReader -Dtests.seed=2EE02ABE17F63E4B
>> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LB
>> -Dtests.timezone=America/Boa_Vista -Dtests.file.encoding=UTF-8
>> [junit4:junit4] ERROR 0.00s J0 | TestFieldsReader (suite) <<<
>> [junit4:junit4] > Throwable #1:
>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0,
>> state=RUNNABLE, group=TGRP-TestFieldsReader]
>> [junit4:junit4] > Caused by:
>> org.apache.lucene.index.MergePolicy$MergeException:
>> java.lang.OutOfMemoryError: Java heap space
>> [junit4:junit4] > at
>> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
>> [junit4:junit4] > at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
>> [junit4:junit4] > at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
>> [junit4:junit4] > Caused by: java.lang.OutOfMemoryError: Java heap
>> space
>> [junit4:junit4] > at
>> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
>> [junit4:junit4] > at
>> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
>> [junit4:junit4] > at
>> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
>> [junit4:junit4] > at
>> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
>> [junit4:junit4] > at
>> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
>> [junit4:junit4] > at
>> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
>> [junit4:junit4] > at
>> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
>> [junit4:junit4] > at
>> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
>> [junit4:junit4] > at
>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
>> [junit4:junit4] > at
>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
>> [junit4:junit4] > at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
>> [junit4:junit4] > at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
>> [junit4:junit4] > at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
>> [junit4:junit4] > at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
>> [junit4:junit4] > at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>> [junit4:junit4] Completed on J0 in 1.73s, 2 tests, 2 errors <<< FAILURES!
>>
>> [...truncated 984 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890:
>> There were test failures: 364 suites, 2312 tests, 1 suite-level error, 1
>> error, 47 ignored (34 assumptions)
>>
>> Total time: 9 minutes 17 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6128 - Still Failing!
Posted by Robert Muir <rc...@gmail.com>.
This has failed several times, since LUCENE-5038 was committed. But that
commit didn't really change the test, except to setCFSRatio(0.0)
and this test indexes only one document!!!!!!!!!!
I'll look at this later tonight if nobody beats me to it, seems like
something is really wrong.
On Tue, Jun 18, 2013 at 4:21 PM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6128/
> Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
>
> 2 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.lucene.index.TestFieldsReader
>
> Error Message:
> Captured an uncaught exception in thread: Thread[id=298, name=Lucene Merge
> Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader]
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0,
> state=RUNNABLE, group=TGRP-TestFieldsReader]
> Caused by: org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: Java heap space
> at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
> at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
> at
> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
> at
> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
> at
> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
> at
> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
> at
> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
> at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
>
>
> REGRESSION: org.apache.lucene.index.TestFieldsReader.testExceptions
>
> Error Message:
> this writer hit an OutOfMemoryError; cannot complete forceMerge
>
> Stack Trace:
> java.lang.IllegalStateException: this writer hit an OutOfMemoryError;
> cannot complete forceMerge
> at
> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
> at
> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
> at
> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
> at
> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:491)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:724)
>
>
>
>
> Build Log:
> [...truncated 395 lines...]
> [junit4:junit4] Suite: org.apache.lucene.index.TestFieldsReader
> [junit4:junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestFieldsReader -Dtests.method=testExceptions
> -Dtests.seed=2EE02ABE17F63E4B -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ar_LB -Dtests.timezone=America/Boa_Vista
> -Dtests.file.encoding=UTF-8
> [junit4:junit4] ERROR 1.28s J0 | TestFieldsReader.testExceptions <<<
> [junit4:junit4] > Throwable #1: java.lang.IllegalStateException: this
> writer hit an OutOfMemoryError; cannot complete forceMerge
> [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
> [junit4:junit4] > at
> org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> [junit4:junit4] 2> WARNING: Will linger awaiting termination of 1 leaked
> thread(s).
> [junit4:junit4] 2> Jun 18, 2013 11:14:11 PM
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
> [junit4:junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene
> Merge Thread #0,6,TGRP-TestFieldsReader]
> [junit4:junit4] 2> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: Java heap space
> [junit4:junit4] 2> at
> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
> [junit4:junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> [junit4:junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> [junit4:junit4] 2> Caused by: java.lang.OutOfMemoryError: Java heap space
> [junit4:junit4] 2> at
> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
> [junit4:junit4] 2> at
> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
> [junit4:junit4] 2> at
> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
> [junit4:junit4] 2> at
> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
> [junit4:junit4] 2> at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
> [junit4:junit4] 2> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
> [junit4:junit4] 2> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
> [junit4:junit4] 2> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
> [junit4:junit4] 2> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
> [junit4:junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> [junit4:junit4] 2> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> [junit4:junit4] 2>
> [junit4:junit4] 2> NOTE: test params are: codec=Lucene3x,
> sim=DefaultSimilarity, locale=ar_LB, timezone=America/Boa_Vista
> [junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation
> 1.8.0-ea (32-bit)/cpus=8,threads=1,free=505999600,total=520093696
> [junit4:junit4] 2> NOTE: All tests run in this JVM:
> [TestLucene40PostingsFormat, TestSimilarity2, Test2BPostingsBytes,
> TestDemo, TestStressAdvance, TestIndexWriterForceMerge, TestArrayUtil,
> TestDocTermOrds, TestSentinelIntSet, TestFieldCacheSanityChecker,
> TestDateTools, TestIndexCommit, TestLucene40PostingsReader,
> TestCachingCollector, TestShardSearching, TestStressNRT,
> TestNoMergeScheduler, TestTopFieldCollector, TestIntroSorter,
> TestNRTThreads, TestCompressingTermVectorsFormat, TestTermInfosReaderIndex,
> TestFastDecompressionMode, TestPayloadTermQuery, TestFieldsReader]
> [junit4:junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestFieldsReader -Dtests.seed=2EE02ABE17F63E4B
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LB
> -Dtests.timezone=America/Boa_Vista -Dtests.file.encoding=UTF-8
> [junit4:junit4] ERROR 0.00s J0 | TestFieldsReader (suite) <<<
> [junit4:junit4] > Throwable #1:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0,
> state=RUNNABLE, group=TGRP-TestFieldsReader]
> [junit4:junit4] > Caused by:
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.OutOfMemoryError: Java heap space
> [junit4:junit4] > at
> __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
> [junit4:junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
> [junit4:junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
> [junit4:junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
> [junit4:junit4] > at
> org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
> [junit4:junit4] > at
> org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
> [junit4:junit4] > at
> org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
> [junit4:junit4] > at
> org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
> [junit4:junit4] > at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
> [junit4:junit4] > at
> org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
> [junit4:junit4] > at
> org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
> [junit4:junit4] > at
> org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
> [junit4:junit4] > at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
> [junit4:junit4] > at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
> [junit4:junit4] > at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
> [junit4:junit4] > at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
> [junit4:junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
> [junit4:junit4] > at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
> [junit4:junit4] Completed on J0 in 1.73s, 2 tests, 2 errors <<< FAILURES!
>
> [...truncated 984 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890:
> There were test failures: 364 suites, 2312 tests, 1 suite-level error, 1
> error, 47 ignored (34 assumptions)
>
> Total time: 9 minutes 17 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6128 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6128/
Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.lucene.index.TestFieldsReader
Error Message:
Captured an uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
at org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
at org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
at org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
at org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
at org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
REGRESSION: org.apache.lucene.index.TestFieldsReader.testExceptions
Error Message:
this writer hit an OutOfMemoryError; cannot complete forceMerge
Stack Trace:
java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot complete forceMerge
at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
at org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 395 lines...]
[junit4:junit4] Suite: org.apache.lucene.index.TestFieldsReader
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFieldsReader -Dtests.method=testExceptions -Dtests.seed=2EE02ABE17F63E4B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LB -Dtests.timezone=America/Boa_Vista -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 1.28s J0 | TestFieldsReader.testExceptions <<<
[junit4:junit4] > Throwable #1: java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot complete forceMerge
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B:58E17816BF8B40FD]:0)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1704)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
[junit4:junit4] > at org.apache.lucene.index.TestFieldsReader.testExceptions(TestFieldsReader.java:204)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> Jun 18, 2013 11:14:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4:junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> Jun 18, 2013 11:14:11 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
[junit4:junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #0,6,TGRP-TestFieldsReader]
[junit4:junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] 2> at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
[junit4:junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
[junit4:junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
[junit4:junit4] 2> Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] 2> at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
[junit4:junit4] 2> at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
[junit4:junit4] 2> at org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
[junit4:junit4] 2> at org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
[junit4:junit4] 2> at org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
[junit4:junit4] 2> at org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
[junit4:junit4] 2> at org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
[junit4:junit4] 2> at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
[junit4:junit4] 2> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
[junit4:junit4] 2> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
[junit4:junit4] 2> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
[junit4:junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
[junit4:junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
[junit4:junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
[junit4:junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=ar_LB, timezone=America/Boa_Vista
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=505999600,total=520093696
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestLucene40PostingsFormat, TestSimilarity2, Test2BPostingsBytes, TestDemo, TestStressAdvance, TestIndexWriterForceMerge, TestArrayUtil, TestDocTermOrds, TestSentinelIntSet, TestFieldCacheSanityChecker, TestDateTools, TestIndexCommit, TestLucene40PostingsReader, TestCachingCollector, TestShardSearching, TestStressNRT, TestNoMergeScheduler, TestTopFieldCollector, TestIntroSorter, TestNRTThreads, TestCompressingTermVectorsFormat, TestTermInfosReaderIndex, TestFastDecompressionMode, TestPayloadTermQuery, TestFieldsReader]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFieldsReader -Dtests.seed=2EE02ABE17F63E4B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LB -Dtests.timezone=America/Boa_Vista -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | TestFieldsReader (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=298, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestFieldsReader]
[junit4:junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([2EE02ABE17F63E4B]:0)
[junit4:junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:541)
[junit4:junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:514)
[junit4:junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4:junit4] > at org.apache.lucene.util.BytesRef.copyBytes(BytesRef.java:196)
[junit4:junit4] > at org.apache.lucene.util.BytesRef.deepCopyOf(BytesRef.java:343)
[junit4:junit4] > at org.apache.lucene.codecs.lucene3x.TermBuffer.toTerm(TermBuffer.java:113)
[junit4:junit4] > at org.apache.lucene.codecs.lucene3x.SegmentTermEnum.term(SegmentTermEnum.java:184)
[junit4:junit4] > at org.apache.lucene.codecs.lucene3x.Lucene3xFields$PreTermsEnum.next(Lucene3xFields.java:863)
[junit4:junit4] > at org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:292)
[junit4:junit4] > at org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:318)
[junit4:junit4] > at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:103)
[junit4:junit4] > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
[junit4:junit4] > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
[junit4:junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3767)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3371)
[junit4:junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:401)
[junit4:junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:478)
[junit4:junit4] Completed on J0 in 1.73s, 2 tests, 2 errors <<< FAILURES!
[...truncated 984 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 364 suites, 2312 tests, 1 suite-level error, 1 error, 47 ignored (34 assumptions)
Total time: 9 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure